2025-03-05 20:29:35,999 [ 172850 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2025-03-05 20:29:35,999 [ 172850 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2025-03-05 20:29:35,999 [ 172850 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2025-03-05 20:29:35,999 [ 172850 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_u7fdr2 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=2993bc2bf171 -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_filesystem_cache/test.py::test_caches_with_the_same_configuration_2[node_caches_with_same_path]' test_filesystem_cache/test.py::test_custom_cached_disk test_filesystem_cache/test.py::test_force_filesystem_cache_on_merges 'test_filesystem_cache/test.py::test_parallel_cache_loading_on_startup[node]' test_filesystem_cache/test.py::test_system_sync_filesystem_cache test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 test_http_native/test.py::test_http_native_returns_timezone test_intersecting_parts/test.py::test_intersect_parts_when_restart test_jbod_balancer/test.py::test_jbod_balanced_merge test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch test_join_set_family_s3/test.py::test_join_s3 test_join_set_family_s3/test.py::test_set_s3 test_kafka_bad_messages/test.py::test_bad_messages_parsing_exception test_kafka_bad_messages/test.py::test_bad_messages_parsing_stream test_kafka_bad_messages/test.py::test_bad_messages_to_mv test_keeper_back_to_back/test.py::test_concurrent_watches test_keeper_back_to_back/test.py::test_end_of_session test_keeper_back_to_back/test.py::test_end_of_watches_session test_keeper_back_to_back/test.py::test_multitransactions test_keeper_back_to_back/test.py::test_random_requests test_keeper_back_to_back/test.py::test_sequential_nodes test_keeper_back_to_back/test.py::test_simple_commands test_keeper_back_to_back/test.py::test_stats test_keeper_back_to_back/test.py::test_watchers test_keeper_client/test.py::test_base_commands test_keeper_client/test.py::test_big_family test_keeper_client/test.py::test_delete_stale_backups test_keeper_client/test.py::test_find_super_nodes test_keeper_client/test.py::test_four_letter_word_commands test_keeper_client/test.py::test_get_all_children_number test_keeper_client/test.py::test_quoted_argument_parsing test_keeper_client/test.py::test_rm_with_version test_keeper_client/test.py::test_rm_without_version test_keeper_client/test.py::test_set_with_version test_keeper_client/test.py::test_set_without_version test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test.py::test_cmd_ydld test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_reconfig_replace_leader_in_one_command/test.py::test_reconfig_replace_leader_in_one_command test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_logs_level/test.py::test_check_client_logs_level test_mask_sensitive_info/test.py::test_backup_table test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size test_memory_limit_observer/test.py::test_observe_memory_limit test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed test_modify_engine_on_restart/test_unusual_path.py::test_modify_engine_on_restart_with_unusual_path test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.2.0, repeat-0.9.3, reportlog-0.4.0, order-1.0.1, random-0.2, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_log_family_s3/test.py::test_log_family_s3[Log] test_keeper_client/test.py::test_base_commands test_filesystem_cache/test.py::test_caches_with_the_same_configuration_2[node_caches_with_same_path] test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_keeper_four_word_command/test.py::test_cmd_clrs test_merge_table_over_distributed/test.py::test_filtering test_kafka_bad_messages/test.py::test_bad_messages_parsing_exception test_keeper_back_to_back/test.py::test_concurrent_watches test_multi_access_storage_role_management/test.py::test_quotas test_mask_sensitive_info/test.py::test_backup_table [gw9] [ 1%] PASSED test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in [gw9] [ 2%] PASSED test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed [gw0] [ 3%] PASSED test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf [gw0] [ 4%] PASSED test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons [gw5] [ 5%] PASSED test_filesystem_cache/test.py::test_caches_with_the_same_configuration_2[node_caches_with_same_path] test_filesystem_cache/test.py::test_custom_cached_disk [gw0] [ 6%] PASSED test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst [gw0] [ 7%] PASSED test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp [gw0] [ 8%] PASSED test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump [gw9] [ 9%] PASSED test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed [gw0] [ 10%] PASSED test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro [gw0] [ 11%] PASSED test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif [gw0] [ 12%] PASSED test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr [gw8] [ 13%] PASSED test_log_family_s3/test.py::test_log_family_s3[Log] test_log_family_s3/test.py::test_log_family_s3[StripeLog] [gw1] [ 14%] PASSED test_keeper_client/test.py::test_base_commands test_keeper_client/test.py::test_big_family [gw1] [ 15%] PASSED test_keeper_client/test.py::test_big_family test_keeper_client/test.py::test_delete_stale_backups [gw0] [ 16%] PASSED test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld [gw1] [ 17%] PASSED test_keeper_client/test.py::test_delete_stale_backups test_keeper_client/test.py::test_find_super_nodes [gw1] [ 18%] PASSED test_keeper_client/test.py::test_find_super_nodes test_keeper_client/test.py::test_four_letter_word_commands [gw1] [ 19%] PASSED test_keeper_client/test.py::test_four_letter_word_commands test_keeper_client/test.py::test_get_all_children_number [gw1] [ 20%] PASSED test_keeper_client/test.py::test_get_all_children_number test_keeper_client/test.py::test_quoted_argument_parsing [gw1] [ 21%] PASSED test_keeper_client/test.py::test_quoted_argument_parsing test_keeper_client/test.py::test_rm_with_version [gw1] [ 22%] PASSED test_keeper_client/test.py::test_rm_with_version test_keeper_client/test.py::test_rm_without_version [gw1] [ 23%] PASSED test_keeper_client/test.py::test_rm_without_version test_keeper_client/test.py::test_set_with_version [gw1] [ 24%] PASSED test_keeper_client/test.py::test_set_with_version test_keeper_client/test.py::test_set_without_version [gw1] [ 25%] PASSED test_keeper_client/test.py::test_set_without_version [gw0] [ 26%] PASSED test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok [gw0] [ 27%] PASSED test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst [gw0] [ 28%] PASSED test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr [gw6] [ 29%] PASSED test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages [gw8] [ 30%] PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] test_log_family_s3/test.py::test_log_family_s3[TinyLog] [gw0] [ 31%] PASSED test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat [gw0] [ 32%] PASSED test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc [gw0] [ 33%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp [gw3] [ 34%] PASSED test_mask_sensitive_info/test.py::test_backup_table test_mask_sensitive_info/test.py::test_backup_to_s3 [gw0] [ 35%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs [gw0] [ 36%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test.py::test_cmd_ydld [gw0] [ 37%] PASSED test_keeper_four_word_command/test.py::test_cmd_ydld [gw8] [ 38%] PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] test_jbod_balancer/test.py::test_jbod_balanced_merge test_keeper_session/test.py::test_session_close_shutdown [gw4] [ 39%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size [gw3] [ 40%] PASSED test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user [gw4] [ 41%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] [gw6] [ 42%] PASSED test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles [gw4] [ 43%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] [gw3] [ 44%] PASSED test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database [gw4] [ 45%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] [gw6] [ 46%] PASSED test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles [gw3] [ 47%] PASSED test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary [gw4] [ 48%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw5] [ 49%] PASSED test_filesystem_cache/test.py::test_custom_cached_disk test_filesystem_cache/test.py::test_force_filesystem_cache_on_merges [gw6] [ 50%] PASSED test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users [gw3] [ 51%] PASSED test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table [gw7] [ 52%] PASSED test_kafka_bad_messages/test.py::test_bad_messages_parsing_exception test_kafka_bad_messages/test.py::test_bad_messages_parsing_stream [gw4] [ 53%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw6] [ 54%] PASSED test_multi_access_storage_role_management/test.py::test_users test_intersecting_parts/test.py::test_intersect_parts_when_restart [gw2] [ 55%] PASSED test_keeper_back_to_back/test.py::test_concurrent_watches test_keeper_back_to_back/test.py::test_end_of_session [gw2] [ 56%] PASSED test_keeper_back_to_back/test.py::test_end_of_session test_keeper_back_to_back/test.py::test_end_of_watches_session test_join_set_family_s3/test.py::test_join_s3 [gw2] [ 57%] PASSED test_keeper_back_to_back/test.py::test_end_of_watches_session test_keeper_back_to_back/test.py::test_multitransactions [gw2] [ 58%] PASSED test_keeper_back_to_back/test.py::test_multitransactions test_keeper_back_to_back/test.py::test_random_requests test_http_native/test.py::test_http_native_returns_timezone [gw1] [ 59%] PASSED test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout [gw1] [ 60%] PASSED test_keeper_session/test.py::test_session_timeout test_recovery_replica/test.py::test_choose_source_replica [gw0] [ 61%] PASSED test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size test_memory_limit_observer/test.py::test_observe_memory_limit [gw4] [ 62%] PASSED test_http_native/test.py::test_http_native_returns_timezone [gw3] [ 63%] PASSED test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions test_keeper_force_recovery_single_node/test.py::test_cluster_recovery [gw0] [ 64%] PASSED test_memory_limit_observer/test.py::test_observe_memory_limit [gw6] [ 65%] PASSED test_intersecting_parts/test.py::test_intersect_parts_when_restart [gw3] [ 66%] PASSED test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster [gw9] [ 67%] PASSED test_jbod_balancer/test.py::test_jbod_balanced_merge test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 [gw3] [ 68%] PASSED test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call [gw8] [ 69%] PASSED test_join_set_family_s3/test.py::test_join_s3 test_join_set_family_s3/test.py::test_set_s3 test_keeper_four_word_command/test_allow_list.py::test_allow_list [gw3] [ 70%] PASSED test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions [gw2] [ 71%] PASSED test_keeper_back_to_back/test.py::test_random_requests test_keeper_back_to_back/test.py::test_sequential_nodes [gw2] [ 72%] PASSED test_keeper_back_to_back/test.py::test_sequential_nodes test_keeper_back_to_back/test.py::test_simple_commands [gw2] [ 73%] PASSED test_keeper_back_to_back/test.py::test_simple_commands test_keeper_back_to_back/test.py::test_stats [gw2] [ 74%] PASSED test_keeper_back_to_back/test.py::test_stats test_keeper_back_to_back/test.py::test_watchers [gw8] [ 75%] PASSED test_join_set_family_s3/test.py::test_set_s3 [gw2] [ 76%] PASSED test_keeper_back_to_back/test.py::test_watchers [gw6] [ 77%] PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list [gw1] [ 78%] PASSED test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_keeper_nodes_move/test.py::test_node_move [gw1] [ 79%] PASSED test_recovery_replica/test.py::test_recovery test_keeper_reconfig_replace_leader_in_one_command/test.py::test_reconfig_replace_leader_in_one_command test_logs_level/test.py::test_check_client_logs_level [gw3] [ 80%] PASSED test_mask_sensitive_info/test.py::test_table_functions [gw1] [ 81%] PASSED test_logs_level/test.py::test_check_client_logs_level test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster [gw9] [ 82%] PASSED test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task [gw3] [ 83%] PASSED test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster [gw4] [ 84%] PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery [gw4] [ 84%] ERROR test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit [gw5] [ 85%] PASSED test_filesystem_cache/test.py::test_force_filesystem_cache_on_merges test_filesystem_cache/test.py::test_parallel_cache_loading_on_startup[node] [gw8] [ 86%] PASSED test_keeper_reconfig_replace_leader_in_one_command/test.py::test_reconfig_replace_leader_in_one_command [gw6] [ 87%] PASSED test_keeper_nodes_move/test.py::test_node_move test_read_temporary_tables_on_failure/test.py::test_different_versions [gw9] [ 88%] PASSED test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table [gw5] [ 89%] PASSED test_filesystem_cache/test.py::test_parallel_cache_loading_on_startup[node] test_filesystem_cache/test.py::test_system_sync_filesystem_cache [gw4] [ 90%] PASSED test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit test_modify_engine_on_restart/test_unusual_path.py::test_modify_engine_on_restart_with_unusual_path [gw8] [ 91%] PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions [gw5] [ 92%] PASSED test_filesystem_cache/test.py::test_system_sync_filesystem_cache [gw9] [ 93%] PASSED test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table [gw0] [ 94%] PASSED test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery [gw4] [ 95%] PASSED test_modify_engine_on_restart/test_unusual_path.py::test_modify_engine_on_restart_with_unusual_path [gw0] [ 96%] PASSED test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery test_part_uuid/test.py::test_part_uuid [gw0] [ 97%] PASSED test_part_uuid/test.py::test_part_uuid [gw7] [ 98%] PASSED test_kafka_bad_messages/test.py::test_bad_messages_parsing_stream test_kafka_bad_messages/test.py::test_bad_messages_to_mv [gw7] [ 99%] PASSED test_kafka_bad_messages/test.py::test_bad_messages_to_mv test_keeper_mntr_pressure/test.py::test_aggressive_mntr [gw7] [100%] PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr ==================================== ERRORS ==================================== __________________ ERROR at teardown of test_cluster_recovery __________________ [gw4] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module") def started_cluster(): try: cluster.start() yield cluster finally: > cluster.shutdown() test_keeper_force_recovery_single_node/test.py:41: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , kill = True ignore_fatal = True def shutdown(self, kill=True, ignore_fatal=True): sanitizer_assert_instance = None fatal_log = None if self.up_called: with open(self.docker_logs_path, "w+") as f: try: subprocess.check_call( # STYLE_CHECK_ALLOW_SUBPROCESS_CHECK_CALL self.base_cmd + ["logs"], stdout=f ) except Exception as e: logging.debug("Unable to get logs from docker.") f.seek(0) for line in f: if SANITIZER_SIGN in line: sanitizer_assert_instance = line.split("|")[0].strip() break if kill: try: run_and_check(self.base_cmd + ["stop", "--timeout", "20"]) except Exception as e: logging.debug( "Kill command failed during shutdown. {}".format(repr(e)) ) logging.debug("Trying to kill forcefully") run_and_check(self.base_cmd + ["kill"]) # Check server logs for Fatal messages and sanitizer failures. # NOTE: we cannot do this via docker since in case of Fatal message container may already die. for name, instance in self.instances.items(): if instance.contains_in_log( SANITIZER_SIGN, from_host=True, filename="stderr.log" ): sanitizer_assert_instance = instance.grep_in_log( SANITIZER_SIGN, from_host=True, filename="stderr.log", after=1000, ) logging.error( "Sanitizer in instance %s log %s", name, sanitizer_assert_instance, ) if not ignore_fatal and instance.contains_in_log( "Fatal", from_host=True ): fatal_log = instance.grep_in_log("Fatal", from_host=True) if "Child process was terminated by signal 9 (KILL)" in fatal_log: fatal_log = None continue logging.error("Crash in instance %s fatal log %s", name, fatal_log) try: subprocess_check_call(self.base_cmd + ["down", "--volumes"]) except Exception as e: logging.debug( "Down + remove orphans failed during shutdown. {}".format(repr(e)) ) else: logging.warning( "docker-compose up was not called. Trying to export docker.log for running containers" ) self.cleanup() self.is_up = False self.docker_client = None for instance in list(self.instances.values()): instance.docker_client = None instance.ip_address = None instance.client = None if sanitizer_assert_instance is not None: > raise Exception( "Sanitizer assert found for instance {}".format( sanitizer_assert_instance ) ) E Exception: Sanitizer assert found for instance ================== E WARNING: ThreadSanitizer: data race (pid=849) E Read of size 1 at 0x7b2000015c10 by thread T582 (mutexes: write M0): E #0 boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:153:15 (clickhouse+0x1ce22a3e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #1 boost::asio::basic_waitable_timer, boost::asio::any_io_executor>::cancel() build_docker/./contrib/boost/boost/asio/basic_waitable_timer.hpp:415:41 (clickhouse+0x1ce22a3e) E #2 nuraft::asio_service::cancel_impl(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/asio_service.cxx:1925:71 (clickhouse+0x1ce22a3e) E #3 nuraft::delayed_task_scheduler::cancel(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/delayed_task_scheduler.hxx:36:9 (clickhouse+0x1ceff126) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #4 nuraft::raft_server::cancel_task(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:367:17 (clickhouse+0x1ceff126) E #5 nuraft::raft_server::restart_election_timer() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:177:9 (clickhouse+0x1ceff126) E #6 nuraft::raft_server::handle_append_entries(nuraft::req_msg&) build_docker/./contrib/NuRaft/src/handle_append_entries.cxx:855:9 (clickhouse+0x1cf0890d) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #7 nuraft::raft_server::process_req(nuraft::req_msg&, nuraft::raft_server::req_ext_params const&) build_docker/./contrib/NuRaft/src/raft_server.cxx:706:16 (clickhouse+0x1cebe4d3) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #8 nuraft::raft_server_handler::process_req(nuraft::raft_server*, nuraft::req_msg&, nuraft::raft_server::req_ext_params const&) build_docker/./contrib/NuRaft/include/libnuraft/raft_server_handler.hxx:42:21 (clickhouse+0x1ce99615) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #9 nuraft::rpc_session::read_complete(std::__1::shared_ptr, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:591:30 (clickhouse+0x1ce99615) E #10 nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)::operator()(boost::system::error_code const&, unsigned long) const build_docker/./contrib/NuRaft/src/asio_service.cxx:395:23 (clickhouse+0x1ce91655) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #11 boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>::operator()(boost::system::error_code, unsigned long, int) build_docker/./contrib/boost/boost/asio/impl/read.hpp:397:9 (clickhouse+0x1ceb1c8b) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #12 boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>::operator()() build_docker/./contrib/boost/boost/asio/detail/bind_handler.hpp:289:5 (clickhouse+0x1ceb1c8b) E #13 void boost::asio::asio_handler_invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, ...) build_docker/./contrib/boost/boost/asio/handler_invoke_hook.hpp:88:3 (clickhouse+0x1ceb17f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #14 void boost_asio_handler_invoke_helpers::invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)&) build_docker/./contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x1ceb17f7) E #15 void boost::asio::detail::asio_handler_invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>*) build_docker/./contrib/boost/boost/asio/impl/read.hpp:465:5 (clickhouse+0x1ceb17f7) E #16 void boost_asio_handler_invoke_helpers::invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>&) build_docker/./contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x1ceb17f7) E #17 void boost::asio::detail::handler_work, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor, void>::complete, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>&) build_docker/./contrib/boost/boost/asio/detail/handler_work.hpp:524:7 (clickhouse+0x1ceb17f7) E #18 boost::asio::detail::reactive_socket_recv_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/reactive_socket_recv_op.hpp:152:9 (clickhouse+0x1ceb17f7) E #19 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x1ce469f2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #20 boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/impl/epoll_reactor.ipp:826:11 (clickhouse+0x1ce469f2) E #21 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x1ce29f8e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #22 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) build_docker/./contrib/boost/boost/asio/detail/impl/scheduler.ipp:493:12 (clickhouse+0x1ce29f8e) E #23 boost::asio::detail::scheduler::run(boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/impl/scheduler.ipp:210:10 (clickhouse+0x1ce296e4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #24 boost::asio::io_context::run() build_docker/./contrib/boost/boost/asio/impl/io_context.ipp:64:24 (clickhouse+0x1ce20d1f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #25 nuraft::asio_service_impl::worker_entry() build_docker/./contrib/NuRaft/src/asio_service.cxx:1807:21 (clickhouse+0x1ce20d1f) E #26 decltype(*std::declval().*std::declval()()) std::__1::__invoke[abi:v15000](void (nuraft::asio_service_impl::*&)(), nuraft::asio_service_impl*&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:359:23 (clickhouse+0x1ce347fb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #27 std::__1::__bind_return, std::__1::tuple<>, __is_valid_bind_return, std::__1::tuple<>>::value>::type std::__1::__apply_functor[abi:v15000], 0ul, std::__1::tuple<>>(void (nuraft::asio_service_impl::*&)(), std::__1::tuple&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/bind.h:263:12 (clickhouse+0x1ce347fb) E #28 std::__1::__bind_return, std::__1::tuple<>, __is_valid_bind_return, std::__1::tuple<>>::value>::type std::__1::__bind::operator()[abi:v15000]<>() build_docker/./contrib/llvm-project/libcxx/include/__functional/bind.h:295:20 (clickhouse+0x1ce347fb) E #29 decltype(std::declval>()()) std::__1::__invoke[abi:v15000]>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1ce347fb) E #30 void std::__1::__thread_execute[abi:v15000]>, std::__1::__bind>(std::__1::tuple>, std::__1::__bind>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0x1ce347fb) E #31 void* std::__1::__thread_proxy[abi:v15000]>, std::__1::__bind>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0x1ce347fb) E E Previous write of size 1 at 0x7b2000015c10 by main thread (mutexes: write M1): E #0 boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:163:35 (clickhouse+0x1ce22a7f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #1 boost::asio::basic_waitable_timer, boost::asio::any_io_executor>::cancel() build_docker/./contrib/boost/boost/asio/basic_waitable_timer.hpp:415:41 (clickhouse+0x1ce22a7f) E #2 nuraft::asio_service::cancel_impl(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/asio_service.cxx:1925:71 (clickhouse+0x1ce22a7f) E #3 nuraft::delayed_task_scheduler::cancel(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/delayed_task_scheduler.hxx:36:9 (clickhouse+0x1cf006e9) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #4 nuraft::raft_server::cancel_task(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:367:17 (clickhouse+0x1cf006e9) E #5 nuraft::raft_server::cancel_schedulers() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:336:9 (clickhouse+0x1cf006e9) E #6 nuraft::raft_server::shutdown() build_docker/./contrib/NuRaft/src/raft_server.cxx:470:5 (clickhouse+0x1cebc41f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #7 DB::KeeperServer::shutdownRaftServer() build_docker/./src/Coordination/KeeperServer.cpp:543:20 (clickhouse+0x19e6a740) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #8 DB::KeeperServer::shutdown() build_docker/./src/Coordination/KeeperServer.cpp:579:5 (clickhouse+0x19e6b3f9) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #9 DB::KeeperDispatcher::shutdown() build_docker/./src/Coordination/KeeperDispatcher.cpp:598:21 (clickhouse+0x19e44dba) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #10 DB::Context::shutdownKeeperDispatcher() const build_docker/./src/Interpreters/Context.cpp:3467:36 (clickhouse+0x16829315) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #11 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1::operator()() const build_docker/./programs/server/Server.cpp:755:5 (clickhouse+0xf1b676c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #12 BasicScopeGuard, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1>::invoke() build_docker/./base/base/../base/scope_guard.h:101:9 (clickhouse+0xf1b676c) E #13 BasicScopeGuard, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1>::~BasicScopeGuard() build_docker/./base/base/../base/scope_guard.h:50:26 (clickhouse+0xf1b676c) E #14 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2167:1 (clickhouse+0xf1a541b) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #15 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #16 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #17 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #18 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #19 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E E Location is heap block of size 120 at 0x7b2000015c00 allocated by main thread: E #0 operator new(unsigned long) (clickhouse+0x70a36f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #1 nuraft::asio_service::schedule(std::__1::shared_ptr&, int) build_docker/./contrib/NuRaft/src/asio_service.cxx:1907:33 (clickhouse+0x1ce22613) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #2 nuraft::raft_server::schedule_task(std::__1::shared_ptr&, int) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:361:21 (clickhouse+0x1ceff41c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #3 nuraft::raft_server::restart_election_timer() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:187:5 (clickhouse+0x1ceff41c) E #4 nuraft::raft_server::start_server(bool) build_docker/./contrib/NuRaft/src/raft_server.cxx:306:9 (clickhouse+0x1cebabff) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #5 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:498:20 (clickhouse+0x19e67d5c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #6 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #7 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #8 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #9 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #10 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #11 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #12 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #13 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #14 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E E Mutex M0 (0x7b6c000049c8) created at: E #0 pthread_mutex_init (clickhouse+0x701ee90) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #1 std::__1::__libcpp_recursive_mutex_init[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:269:10 (clickhouse+0x202fba84) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #2 std::__1::recursive_mutex::recursive_mutex() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:61:14 (clickhouse+0x202fba84) E #3 nuraft::raft_server::raft_server(nuraft::context*, nuraft::raft_server::init_options const&) build_docker/./contrib/NuRaft/src/raft_server.cxx:50:14 (clickhouse+0x1ceb5eec) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #4 DB::KeeperServer::KeeperRaftServer::KeeperRaftServer(nuraft::context*, nuraft::raft_server::init_options const&) build_docker/./src/Coordination/KeeperServer.cpp:279:32 (clickhouse+0x19e82f10) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #5 DB::KeeperServer::KeeperRaftServer* std::__1::construct_at[abi:v15000](DB::KeeperServer::KeeperRaftServer*, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x19e82f10) E #6 void std::__1::allocator_traits>::construct[abi:v15000](std::__1::allocator&, DB::KeeperServer::KeeperRaftServer*, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x19e82f10) E #7 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000](std::__1::allocator, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x19e82f10) E #8 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], nuraft::context*&, nuraft::raft_server::init_options&, void>(std::__1::allocator const&, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x19e82f10) E #9 std::__1::shared_ptr std::__1::make_shared[abi:v15000](nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19e67a43) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #10 std::__1::shared_ptr nuraft::cs_new(nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x19e67a43) E #11 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:487:21 (clickhouse+0x19e67a43) E #12 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #13 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #14 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #15 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #17 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #20 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E E Mutex M1 (0x7b9800002be8) created at: E #0 pthread_mutex_lock (clickhouse+0x701f1ab) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #1 std::__1::__libcpp_mutex_lock[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:304:10 (clickhouse+0x202fb979) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #2 std::__1::mutex::lock() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:38:14 (clickhouse+0x202fb979) E #3 std::__1::lock_guard::lock_guard[abi:v15000](std::__1::mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:94:27 (clickhouse+0x16829417) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #4 DB::Context::updateKeeperConfiguration(Poco::Util::AbstractConfiguration const&) build_docker/./src/Interpreters/Context.cpp:3477:21 (clickhouse+0x16829417) E #5 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6::operator()(Poco::AutoPtr, bool) const build_docker/./programs/server/Server.cpp:1457:33 (clickhouse+0xf1c0c06) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #6 decltype(std::declval, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&>()(std::declval>(), std::declval())) std::__1::__invoke[abi:v15000], std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr, bool>(DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf1bef49) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #7 void std::__1::__invoke_void_return_wrapper::__call, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr, bool>(DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf1bef49) E #8 std::__1::__function::__default_alloc_func, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, void (Poco::AutoPtr, bool)>::operator()[abi:v15000](Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf1bef49) E #9 void std::__1::__function::__policy_invoker, bool)>::__call_impl, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, void (Poco::AutoPtr, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf1bef49) E #10 std::__1::__function::__policy_func, bool)>::operator()[abi:v15000](Poco::AutoPtr&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fae13e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #11 std::__1::function, bool)>::operator()(Poco::AutoPtr, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fae13e) E #12 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fae13e) E #13 DB::ConfigReloader::ConfigReloader(std::__1::basic_string_view>, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&, std::__1::basic_string, std::__1::allocator> const&, zkutil::ZooKeeperNodeCache&&, std::__1::shared_ptr const&, std::__1::function, bool)>&&, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:32:9 (clickhouse+0x19facb5f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #14 std::__1::__unique_if::__unique_single std::__1::make_unique[abi:v15000], std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::basic_string, std::__1::allocator>, zkutil::ZooKeeperNodeCache, std::__1::shared_ptr&, DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, bool>(std::__1::basic_string, std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::basic_string, std::__1::allocator>&&, zkutil::ZooKeeperNodeCache&&, std::__1::shared_ptr&, DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0xf19f4b8) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #15 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1329:33 (clickhouse+0xf19f4b8) E #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #17 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #20 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E E Thread T582 (tid=1457, running) created by main thread at: E #0 pthread_create (clickhouse+0x701d4ab) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #1 std::__1::__libcpp_thread_create[abi:v15000](unsigned long*, void* (*)(void*), void*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:376:10 (clickhouse+0x1ce346de) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #2 std::__1::thread::thread, void>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/thread:311:16 (clickhouse+0x1ce346de) E #3 std::__1::thread* std::__1::construct_at[abi:v15000], std::__1::thread*>(std::__1::thread*, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x1ce1ef38) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #4 void std::__1::allocator_traits>::construct[abi:v15000], void, void>(std::__1::allocator&, std::__1::thread*, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x1ce1ef38) E #5 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000]>(std::__1::allocator, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x1ce1ef38) E #6 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], std::__1::__bind, void>(std::__1::allocator const&, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x1ce1ef38) E #7 std::__1::shared_ptr std::__1::make_shared[abi:v15000], void>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x1ce1ef38) E #8 std::__1::shared_ptr nuraft::cs_new>(std::__1::__bind&&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x1ce1ef38) E #9 nuraft::asio_service_impl::asio_service_impl(nuraft::asio_service_options const&, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:1768:13 (clickhouse+0x1ce1ef38) E #10 nuraft::asio_service::asio_service(nuraft::asio_service_options const&, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:1897:17 (clickhouse+0x1ce22195) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #11 nuraft::asio_service* std::__1::construct_at[abi:v15000]&, nuraft::asio_service*>(nuraft::asio_service*, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x19e673f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #12 void std::__1::allocator_traits>::construct[abi:v15000]&, void, void>(std::__1::allocator&, nuraft::asio_service*, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x19e673f7) E #13 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000]&>(std::__1::allocator, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x19e673f7) E #14 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], nuraft::asio_service_options&, std::__1::shared_ptr&, void>(std::__1::allocator const&, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x19e673f7) E #15 std::__1::shared_ptr std::__1::make_shared[abi:v15000]&, void>(nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19e673f7) E #16 std::__1::shared_ptr nuraft::cs_new&>(nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x19e673f7) E #17 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:455:20 (clickhouse+0x19e673f7) E #18 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #19 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #20 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #21 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #22 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #23 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #24 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #25 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E #26 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) E E SUMMARY: ThreadSanitizer: data race build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:153:15 in boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) E ================== E ThreadSanitizer: reported 1 warnings E Logging test to /var/log/clickhouse-server/clickhouse-server.log E Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log helpers/cluster.py:3113: Exception ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2025-03-05 20:30:46 [ 554 ] INFO : Running tests in /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/test.py (cluster.py:2658, start) 2025-03-05 20:30:46 [ 554 ] DEBUG : Cluster start called. is_up=False (cluster.py:2665, start) 2025-03-05 20:30:46 [ 554 ] DEBUG : Docker networks for project roottestkeeperforcerecoverysinglenode are NETWORK ID NAME DRIVER SCOPE (cluster.py:776, print_all_docker_pieces) 2025-03-05 20:30:46 [ 554 ] DEBUG : Docker containers for project roottestkeeperforcerecoverysinglenode are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:784, print_all_docker_pieces) 2025-03-05 20:30:46 [ 554 ] DEBUG : Docker volumes for project roottestkeeperforcerecoverysinglenode are DRIVER VOLUME NAME (cluster.py:792, print_all_docker_pieces) 2025-03-05 20:30:46 [ 554 ] DEBUG : Cleanup called (cluster.py:797, cleanup) 2025-03-05 20:30:46 [ 554 ] DEBUG : Docker networks for project roottestkeeperforcerecoverysinglenode are NETWORK ID NAME DRIVER SCOPE (cluster.py:776, print_all_docker_pieces) 2025-03-05 20:30:46 [ 554 ] DEBUG : Docker containers for project roottestkeeperforcerecoverysinglenode are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:784, print_all_docker_pieces) 2025-03-05 20:30:46 [ 554 ] DEBUG : Docker volumes for project roottestkeeperforcerecoverysinglenode are DRIVER VOLUME NAME (cluster.py:792, print_all_docker_pieces) 2025-03-05 20:30:46 [ 554 ] DEBUG : Command:docker container list --all --filter name='^/roottestkeeperforcerecoverysinglenode_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:109, run_and_check) 2025-03-05 20:30:46 [ 554 ] DEBUG : Unstopped containers: {} (cluster.py:811, cleanup) 2025-03-05 20:30:46 [ 554 ] DEBUG : No running containers for project: roottestkeeperforcerecoverysinglenode (cluster.py:825, cleanup) 2025-03-05 20:30:46 [ 554 ] DEBUG : Trying to prune unused networks... (cluster.py:831, cleanup) 2025-03-05 20:30:46 [ 554 ] DEBUG : Trying to prune unused images... (cluster.py:847, cleanup) 2025-03-05 20:30:46 [ 554 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:109, run_and_check) 2025-03-05 20:30:46 [ 554 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:117, run_and_check) 2025-03-05 20:30:46 [ 554 ] DEBUG : Images pruned (cluster.py:850, cleanup) 2025-03-05 20:30:46 [ 554 ] DEBUG : Trying to prune unused volumes... (cluster.py:856, cleanup) 2025-03-05 20:30:46 [ 554 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:109, run_and_check) 2025-03-05 20:30:46 [ 554 ] DEBUG : Stdout:11 (cluster.py:117, run_and_check) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup directory for instance: node1 (cluster.py:2678, start) 2025-03-05 20:30:46 [ 554 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4378, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Create directory for common tests configuration (cluster.py:4383, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Copy common configuration from helpers (cluster.py:4403, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Generate and write macros file (cluster.py:4436, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/configs/enable_keeper1.xml'] to /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/configs/config.d (cluster.py:4466, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/database (cluster.py:4483, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/logs (cluster.py:4494, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4579, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup directory for instance: node2 (cluster.py:2678, start) 2025-03-05 20:30:46 [ 554 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4378, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Create directory for common tests configuration (cluster.py:4383, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Copy common configuration from helpers (cluster.py:4403, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Generate and write macros file (cluster.py:4436, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/configs/enable_keeper2.xml'] to /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/configs/config.d (cluster.py:4466, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/database (cluster.py:4483, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/logs (cluster.py:4494, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4579, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup directory for instance: node3 (cluster.py:2678, start) 2025-03-05 20:30:46 [ 554 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4378, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Create directory for common tests configuration (cluster.py:4383, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Copy common configuration from helpers (cluster.py:4403, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Generate and write macros file (cluster.py:4436, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/configs/enable_keeper3.xml'] to /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/configs/config.d (cluster.py:4466, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/database (cluster.py:4483, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/logs (cluster.py:4494, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4579, create_dir) 2025-03-05 20:30:46 [ 554 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/.env (cluster.py:82, _create_env_file) 2025-03-05 20:30:46 [ 554 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-03-05 20:30:46 [ 554 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-03-05 20:30:46 [ 554 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-03-05 20:30:46 [ 554 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-03-05 20:30:46 [ 554 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2025-03-05 20:30:46 [ 554 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/.env', '--project-name', 'roottestkeeperforcerecoverysinglenode', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/docker-compose.yml', 'pull'] (cluster.py:109, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node1 ... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node3 ... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node2 ... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node1 ... pulling from altinityinfra/integr... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node1 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node1 ... status: image is up to date for a... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node2 ... pulling from altinityinfra/integr... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node2 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node2 ... status: image is up to date for a... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node2 ... done (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node3 ... pulling from altinityinfra/integr... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node3 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node3 ... status: image is up to date for a... (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : Stderr:Pulling node3 ... done (cluster.py:119, run_and_check) 2025-03-05 20:30:57 [ 554 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/.env --project-name roottestkeeperforcerecoverysinglenode --file /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/docker-compose.yml up -d --no-recreate') (cluster.py:3001, start) 2025-03-05 20:30:57 [ 554 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/.env', '--project-name', 'roottestkeeperforcerecoverysinglenode', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:109, run_and_check) 2025-03-05 20:30:59 [ 554 ] DEBUG : Stderr:Creating network "roottestkeeperforcerecoverysinglenode_default" with the default driver (cluster.py:119, run_and_check) 2025-03-05 20:30:59 [ 554 ] DEBUG : Stderr:Creating roottestkeeperforcerecoverysinglenode_node2_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:30:59 [ 554 ] DEBUG : Stderr:Creating roottestkeeperforcerecoverysinglenode_node3_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:30:59 [ 554 ] DEBUG : Stderr:Creating roottestkeeperforcerecoverysinglenode_node1_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:30:59 [ 554 ] DEBUG : Stderr:Creating roottestkeeperforcerecoverysinglenode_node1_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:30:59 [ 554 ] DEBUG : Stderr:Creating roottestkeeperforcerecoverysinglenode_node2_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:30:59 [ 554 ] DEBUG : Stderr:Creating roottestkeeperforcerecoverysinglenode_node3_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:30:59 [ 554 ] DEBUG : ClickHouse instance created (cluster.py:3009, start) 2025-03-05 20:30:59 [ 554 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2004, get_instance_ip) 2025-03-05 20:30:59 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:30:59 [ 554 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.19.2... (cluster.py:3016, start) 2025-03-05 20:30:59 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:30:59 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:30:59 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:30:59 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:30:59 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:30:59 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:00 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:01 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/7485767d82e3940f90383f63a7e5a9e43de75234b537226109ca86492741f50c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : ClickHouse node1 started (cluster.py:3020, start) 2025-03-05 20:31:02 [ 554 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.19.4... (cluster.py:3016, start) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/02131c42f8f48d01ec44aebf7e1b44e6b5c3cb9ebfe931f8d5426fc3aadfe7e6/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : ClickHouse node2 started (cluster.py:3020, start) 2025-03-05 20:31:02 [ 554 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : Waiting for ClickHouse start in node3, ip: 172.16.19.3... (cluster.py:3016, start) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/ff253ebf206bb159bcd8969d7d730bda4c7b3c8fb9a43bc22ed2bd36ffca9613/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : ClickHouse node3 started (cluster.py:3020, start) ------------------------------ Captured log call ------------------------------- 2025-03-05 20:31:02 [ 554 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-03-05 20:31:02 [ 554 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] INFO : Connecting to 172.16.19.4(172.16.19.4):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-03-05 20:31:02 [ 554 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:02 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:02 [ 554 ] INFO : Connecting to 172.16.19.3(172.16.19.3):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_node1', data=b'somedata1', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=1): '/test_force_recovery_node1' (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_node2', data=b'somedata2', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=1): '/test_force_recovery_node2' (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_node3', data=b'somedata3', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=1): '/test_force_recovery_node3' (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=2): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=1, mzxid=1, ctime=1741206662754, mtime=1741206662754, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=3): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=3): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1741206662754, mtime=1741206662754, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=4): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=2, mzxid=2, ctime=1741206662760, mtime=1741206662760, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=5): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=5): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1741206662760, mtime=1741206662760, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=6): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=3, mzxid=3, ctime=1741206662765, mtime=1741206662765, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=7): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=7): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1741206662765, mtime=1741206662765, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=2): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=1, mzxid=1, ctime=1741206662754, mtime=1741206662754, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=3): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=3): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1741206662754, mtime=1741206662754, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=4): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=2, mzxid=2, ctime=1741206662760, mtime=1741206662760, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=5): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=5): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1741206662760, mtime=1741206662760, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=6): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=3, mzxid=3, ctime=1741206662765, mtime=1741206662765, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=7): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=7): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1741206662765, mtime=1741206662765, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=2): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=1, mzxid=1, ctime=1741206662754, mtime=1741206662754, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=3): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=3): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1741206662754, mtime=1741206662754, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=4): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=2, mzxid=2, ctime=1741206662760, mtime=1741206662760, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=5): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=5): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1741206662760, mtime=1741206662760, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=6): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=3, mzxid=3, ctime=1741206662765, mtime=1741206662765, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : Sending request(xid=7): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:02 [ 554 ] DEBUG : Received response(xid=7): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1741206662765, mtime=1741206662765, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-03-05 20:31:02 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:02 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:109, run_and_check) 2025-03-05 20:31:02 [ 554 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:117, run_and_check) 2025-03-05 20:31:02 [ 554 ] DEBUG : Stdout: 8 ? 00:00:01 clickhouse (cluster.py:117, run_and_check) 2025-03-05 20:31:02 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:02 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:109, run_and_check) 2025-03-05 20:31:02 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:02 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:02 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:03 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:03 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:04 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:05 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:05 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:05 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:06 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:06 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:06 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:07 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:07 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:07 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:08 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:08 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:08 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:09 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:09 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:09 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:10 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:10 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:10 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:11 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:11 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:11 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:12 [ 554 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-03-05 20:31:12 [ 554 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-03-05 20:31:12 [ 554 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-03-05 20:31:12 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:12 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:12 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:12 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:12 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:12 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:12 [ 554 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:12 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:12 [ 554 ] INFO : Connecting to 172.16.19.4(172.16.19.4):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:12 [ 554 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-03-05 20:31:12 [ 554 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-03-05 20:31:12 [ 554 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_extra', data=b'somedataextra', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-03-05 20:31:12 [ 554 ] DEBUG : Received response(xid=1): '/test_force_recovery_extra' (connection.py:410, _read_response) 2025-03-05 20:31:12 [ 554 ] DEBUG : Sending request(xid=8): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:12 [ 554 ] DEBUG : Received response(xid=8): ZnodeStat(czxid=13, mzxid=13, ctime=1741206672695, mtime=1741206672695, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=13) (connection.py:410, _read_response) 2025-03-05 20:31:12 [ 554 ] DEBUG : Sending request(xid=9): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:12 [ 554 ] DEBUG : Received response(xid=9): (b'somedataextra', ZnodeStat(czxid=13, mzxid=13, ctime=1741206672695, mtime=1741206672695, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=13)) (connection.py:410, _read_response) 2025-03-05 20:31:12 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:12 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:12 [ 554 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3812, start_clickhouse) 2025-03-05 20:31:12 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:12 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:109, run_and_check) 2025-03-05 20:31:13 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:13 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:13 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:13 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:13 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:13 [ 554 ] DEBUG : Clickhouse process running. (cluster.py:3823, start_clickhouse) 2025-03-05 20:31:13 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:13 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:14 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:14 [ 554 ] DEBUG : Executing query select 20 on node1 (cluster.py:3450, query) 2025-03-05 20:31:14 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:14 [ 554 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=3, time_out=30000, session_id=1, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-03-05 20:31:14 [ 554 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-03-05 20:31:14 [ 554 ] DEBUG : Executing query select 20 on node1 (cluster.py:3450, query) 2025-03-05 20:31:15 [ 554 ] DEBUG : Executing query select 20 on node1 (cluster.py:3450, query) 2025-03-05 20:31:15 [ 554 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:15 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:15 [ 554 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2004, get_instance_ip) 2025-03-05 20:31:15 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:31:15 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:15 [ 554 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-03-05 20:31:15 [ 554 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-03-05 20:31:15 [ 554 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:15 [ 554 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=13, mzxid=13, ctime=1741206672695, mtime=1741206672695, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=13) (connection.py:410, _read_response) 2025-03-05 20:31:15 [ 554 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-03-05 20:31:15 [ 554 ] DEBUG : Received response(xid=2): (b'somedataextra', ZnodeStat(czxid=13, mzxid=13, ctime=1741206672695, mtime=1741206672695, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=13)) (connection.py:410, _read_response) 2025-03-05 20:31:15 [ 554 ] DEBUG : Sending request(xid=3): Close() (connection.py:312, _submit) 2025-03-05 20:31:15 [ 554 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-03-05 20:31:15 [ 554 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-03-05 20:31:15 [ 554 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-03-05 20:31:15 [ 554 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-03-05 20:31:15 [ 554 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-03-05 20:31:15 [ 554 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2025-03-05 20:31:15 [ 554 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-03-05 20:31:15 [ 554 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-03-05 20:31:15 [ 554 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-03-05 20:31:15 [ 554 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-03-05 20:31:15 [ 554 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-03-05 20:31:15 [ 554 ] DEBUG : Sending request(xid=10): Close() (connection.py:312, _submit) 2025-03-05 20:31:15 [ 554 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-03-05 20:31:15 [ 554 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-03-05 20:31:15 [ 554 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-03-05 20:31:15 [ 554 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-03-05 20:31:15 [ 554 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-03-05 20:31:15 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:15 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:109, run_and_check) 2025-03-05 20:31:15 [ 554 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:117, run_and_check) 2025-03-05 20:31:15 [ 554 ] DEBUG : Stdout: 849 ? 00:00:02 clickhouse (cluster.py:117, run_and_check) 2025-03-05 20:31:15 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:15 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:109, run_and_check) 2025-03-05 20:31:16 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:16 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:16 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:17 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:17 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:17 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:18 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:18 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:18 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:19 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:19 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:19 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:20 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:20 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:20 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:21 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:21 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:21 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:22 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:22 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:22 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:23 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:23 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:23 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:24 [ 554 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-03-05 20:31:24 [ 554 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-03-05 20:31:24 [ 554 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-03-05 20:31:24 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:24 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:24 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:24 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:24 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:24 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:24 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:24 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:24 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:25 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:25 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:25 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:25 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:25 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:26 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:26 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:26 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:27 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:27 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:28 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:28 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:28 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:29 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:29 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:29 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:30 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:30 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:30 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:31 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:31 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:31 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:31 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:31 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:32 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:32 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:32 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:33 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:33 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:33 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:34 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:34 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:34 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:35 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:35 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:35 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:36 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:36 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:36 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:37 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:37 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:37 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:38 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:38 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:38 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:39 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:39 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:39 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:39 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:39 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:40 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:40 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:40 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:41 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:41 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:42 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:43 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:43 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:43 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:44 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:44 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:44 [ 554 ] DEBUG : Stdout:849 (cluster.py:117, run_and_check) 2025-03-05 20:31:45 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:45 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:45 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:45 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:109, run_and_check) 2025-03-05 20:31:45 [ 554 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:117, run_and_check) 2025-03-05 20:31:45 [ 554 ] DEBUG : Stdout: 8 ? 00:00:06 clickhouse (cluster.py:117, run_and_check) 2025-03-05 20:31:45 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:45 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:109, run_and_check) 2025-03-05 20:31:45 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:45 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:45 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:46 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:46 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:46 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:47 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:47 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:47 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:48 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:48 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:48 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:49 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:49 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:49 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:50 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:50 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:50 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:51 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:51 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:51 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:52 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:52 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:52 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:53 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:53 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:53 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:54 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:54 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:55 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:55 [ 554 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-03-05 20:31:55 [ 554 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-03-05 20:31:55 [ 554 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-03-05 20:31:55 [ 554 ] INFO : Connecting to 172.16.19.4(172.16.19.4):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:55 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:56 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node2_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:56 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node2_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:56 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node3_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:56 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecoverysinglenode_node3_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:109, run_and_check) 2025-03-05 20:31:56 [ 554 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:117, run_and_check) 2025-03-05 20:31:56 [ 554 ] DEBUG : Stdout: 8 ? 00:00:09 clickhouse (cluster.py:117, run_and_check) 2025-03-05 20:31:56 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node3_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:56 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecoverysinglenode_node3_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:109, run_and_check) 2025-03-05 20:31:56 [ 554 ] INFO : Connecting to 172.16.19.4(172.16.19.4):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:56 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:56 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node3_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:56 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node3_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:56 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:56 [ 554 ] INFO : Connecting to 172.16.19.4(172.16.19.4):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:56 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:57 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node3_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:57 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node3_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:57 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:57 [ 554 ] INFO : Connecting to 172.16.19.4(172.16.19.4):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:57 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:58 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node3_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:58 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node3_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:58 [ 554 ] DEBUG : Stdout:8 (cluster.py:117, run_and_check) 2025-03-05 20:31:58 [ 554 ] INFO : Connecting to 172.16.19.4(172.16.19.4):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:58 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:58 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:31:58 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:31:59 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node3_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:59 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node3_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:59 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDxmb3JjZV9yZWNvdmVyeT4xPC9mb3JjZV9yZWNvdmVyeT4KICAgICAgICA8dGNwX3BvcnQ+OTE4MTwvdGNwX3BvcnQ+CiAgICAgICAgPHNlcnZlcl9pZD4xPC9zZXJ2ZXJfaWQ+CiAgICAgICAgPGxvZ19zdG9yYWdlX3BhdGg+L3Zhci9saWIvY2xpY2tob3VzZS9jb29yZGluYXRpb24vbG9nPC9sb2dfc3RvcmFnZV9wYXRoPgogICAgICAgIDxzbmFwc2hvdF9zdG9yYWdlX3BhdGg+L3Zhci9saWIvY2xpY2tob3VzZS9jb29yZGluYXRpb24vc25hcHNob3RzPC9zbmFwc2hvdF9zdG9yYWdlX3BhdGg+CgogICAgICAgIDxjb29yZGluYXRpb25fc2V0dGluZ3M+CiAgICAgICAgICAgIDxvcGVyYXRpb25fdGltZW91dF9tcz41MDAwPC9vcGVyYXRpb25fdGltZW91dF9tcz4KICAgICAgICAgICAgPHNlc3Npb25fdGltZW91dF9tcz4xMDAwMDwvc2Vzc2lvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c25hcHNob3RfZGlzdGFuY2U+NzU8L3NuYXBzaG90X2Rpc3RhbmNlPgogICAgICAgICAgICA8cmFmdF9sb2dzX2xldmVsPnRyYWNlPC9yYWZ0X2xvZ3NfbGV2ZWw+CiAgICAgICAgICAgIDxtYXhfcmVxdWVzdHNfYmF0Y2hfc2l6ZT4yMDA8L21heF9yZXF1ZXN0c19iYXRjaF9zaXplPgogICAgICAgIDwvY29vcmRpbmF0aW9uX3NldHRpbmdzPgoKICAgICAgICA8cmFmdF9jb25maWd1cmF0aW9uPgogICAgICAgICAgICA8c2VydmVyPgogICAgICAgICAgICAgICAgPGlkPjE8L2lkPgogICAgICAgICAgICAgICAgPGhvc3RuYW1lPm5vZGUxPC9ob3N0bmFtZT4KICAgICAgICAgICAgICAgIDxwb3J0PjkyMzQ8L3BvcnQ+CiAgICAgICAgICAgIDwvc2VydmVyPgogICAgICAgIDwvcmFmdF9jb25maWd1cmF0aW9uPgogICAgPC9rZWVwZXJfc2VydmVyPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper1.xml'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:59 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDxmb3JjZV9yZWNvdmVyeT4xPC9mb3JjZV9yZWNvdmVyeT4KICAgICAgICA8dGNwX3BvcnQ+OTE4MTwvdGNwX3BvcnQ+CiAgICAgICAgPHNlcnZlcl9pZD4xPC9zZXJ2ZXJfaWQ+CiAgICAgICAgPGxvZ19zdG9yYWdlX3BhdGg+L3Zhci9saWIvY2xpY2tob3VzZS9jb29yZGluYXRpb24vbG9nPC9sb2dfc3RvcmFnZV9wYXRoPgogICAgICAgIDxzbmFwc2hvdF9zdG9yYWdlX3BhdGg+L3Zhci9saWIvY2xpY2tob3VzZS9jb29yZGluYXRpb24vc25hcHNob3RzPC9zbmFwc2hvdF9zdG9yYWdlX3BhdGg+CgogICAgICAgIDxjb29yZGluYXRpb25fc2V0dGluZ3M+CiAgICAgICAgICAgIDxvcGVyYXRpb25fdGltZW91dF9tcz41MDAwPC9vcGVyYXRpb25fdGltZW91dF9tcz4KICAgICAgICAgICAgPHNlc3Npb25fdGltZW91dF9tcz4xMDAwMDwvc2Vzc2lvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c25hcHNob3RfZGlzdGFuY2U+NzU8L3NuYXBzaG90X2Rpc3RhbmNlPgogICAgICAgICAgICA8cmFmdF9sb2dzX2xldmVsPnRyYWNlPC9yYWZ0X2xvZ3NfbGV2ZWw+CiAgICAgICAgICAgIDxtYXhfcmVxdWVzdHNfYmF0Y2hfc2l6ZT4yMDA8L21heF9yZXF1ZXN0c19iYXRjaF9zaXplPgogICAgICAgIDwvY29vcmRpbmF0aW9uX3NldHRpbmdzPgoKICAgICAgICA8cmFmdF9jb25maWd1cmF0aW9uPgogICAgICAgICAgICA8c2VydmVyPgogICAgICAgICAgICAgICAgPGlkPjE8L2lkPgogICAgICAgICAgICAgICAgPGhvc3RuYW1lPm5vZGUxPC9ob3N0bmFtZT4KICAgICAgICAgICAgICAgIDxwb3J0PjkyMzQ8L3BvcnQ+CiAgICAgICAgICAgIDwvc2VydmVyPgogICAgICAgIDwvcmFmdF9jb25maWd1cmF0aW9uPgogICAgPC9rZWVwZXJfc2VydmVyPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper1.xml'] (cluster.py:109, run_and_check) 2025-03-05 20:31:59 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:59 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:31:59 [ 554 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3812, start_clickhouse) 2025-03-05 20:31:59 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2037, exec_in_container) 2025-03-05 20:31:59 [ 554 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:109, run_and_check) 2025-03-05 20:32:00 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:32:00 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:32:00 [ 554 ] DEBUG : Stdout:1941 (cluster.py:117, run_and_check) 2025-03-05 20:32:00 [ 554 ] DEBUG : Clickhouse process running. (cluster.py:3823, start_clickhouse) 2025-03-05 20:32:00 [ 554 ] DEBUG : run container_id:roottestkeeperforcerecoverysinglenode_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2037, exec_in_container) 2025-03-05 20:32:00 [ 554 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecoverysinglenode_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:109, run_and_check) 2025-03-05 20:32:00 [ 554 ] DEBUG : Stdout:1941 (cluster.py:117, run_and_check) 2025-03-05 20:32:00 [ 554 ] DEBUG : Executing query select 20 on node1 (cluster.py:3450, query) 2025-03-05 20:32:01 [ 554 ] INFO : Connecting to 172.16.19.4(172.16.19.4):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:32:01 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:32:01 [ 554 ] DEBUG : Executing query select 20 on node1 (cluster.py:3450, query) 2025-03-05 20:32:02 [ 554 ] DEBUG : Executing query select 20 on node1 (cluster.py:3450, query) 2025-03-05 20:32:02 [ 554 ] DEBUG : Executing query select 20 on node1 (cluster.py:3450, query) 2025-03-05 20:32:03 [ 554 ] DEBUG : Executing query select 20 on node1 (cluster.py:3450, query) 2025-03-05 20:32:03 [ 554 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2004, get_instance_ip) 2025-03-05 20:32:03 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:32:03 [ 554 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2004, get_instance_ip) 2025-03-05 20:32:03 [ 554 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecoverysinglenode_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-03-05 20:32:03 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=1, mzxid=1, ctime=1741206662754, mtime=1741206662754, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] DEBUG : Received response(xid=2): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1741206662754, mtime=1741206662754, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=3): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] DEBUG : Received response(xid=3): ZnodeStat(czxid=2, mzxid=2, ctime=1741206662760, mtime=1741206662760, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=4): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] DEBUG : Received response(xid=4): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1741206662760, mtime=1741206662760, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=5): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] DEBUG : Received response(xid=5): ZnodeStat(czxid=3, mzxid=3, ctime=1741206662765, mtime=1741206662765, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=6): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] DEBUG : Received response(xid=6): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1741206662765, mtime=1741206662765, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=7): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] DEBUG : Received response(xid=7): ZnodeStat(czxid=13, mzxid=13, ctime=1741206672695, mtime=1741206672695, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=13) (connection.py:410, _read_response) 2025-03-05 20:32:03 [ 554 ] DEBUG : Sending request(xid=8): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-03-05 20:32:03 [ 554 ] DEBUG : Received response(xid=8): (b'somedataextra', ZnodeStat(czxid=13, mzxid=13, ctime=1741206672695, mtime=1741206672695, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=13)) (connection.py:410, _read_response) ---------------------------- Captured log teardown ----------------------------- 2025-03-05 20:32:04 [ 554 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/.env', '--project-name', 'roottestkeeperforcerecoverysinglenode', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:109, run_and_check) 2025-03-05 20:32:04 [ 554 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-03-05 20:32:04 [ 554 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-03-05 20:32:04 [ 554 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-03-05 20:32:04 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:32:04 [ 554 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-03-05 20:32:04 [ 554 ] INFO : Connecting to 172.16.19.2(172.16.19.2):9181, use_ssl: False (connection.py:650, _connect) 2025-03-05 20:32:04 [ 554 ] DEBUG : Stderr:Stopping roottestkeeperforcerecoverysinglenode_node1_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : Stderr:Stopping roottestkeeperforcerecoverysinglenode_node3_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : Stderr:Stopping roottestkeeperforcerecoverysinglenode_node2_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : Stderr:Stopping roottestkeeperforcerecoverysinglenode_node3_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : Stderr:Stopping roottestkeeperforcerecoverysinglenode_node2_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : Stderr:Stopping roottestkeeperforcerecoverysinglenode_node1_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/logs/stderr.log* || true'] (cluster.py:109, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : Stdout:/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/logs/stderr.log:================== (cluster.py:117, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : Stdout:/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/logs/stderr.log:================== (cluster.py:117, run_and_check) 2025-03-05 20:32:04 [ 554 ] DEBUG : grep in log called ================== (cluster.py:3941, grep_in_log) 2025-03-05 20:32:05 [ 554 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/logs/stderr.log ] && zgrep -A1000 -a "==================" /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/logs/stderr.log* || true'] (cluster.py:109, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout:================== (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout:WARNING: ThreadSanitizer: data race (pid=849) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: Read of size 1 at 0x7b2000015c10 by thread T582 (mutexes: write M0): (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #0 boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:153:15 (clickhouse+0x1ce22a3e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #1 boost::asio::basic_waitable_timer, boost::asio::any_io_executor>::cancel() build_docker/./contrib/boost/boost/asio/basic_waitable_timer.hpp:415:41 (clickhouse+0x1ce22a3e) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #2 nuraft::asio_service::cancel_impl(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/asio_service.cxx:1925:71 (clickhouse+0x1ce22a3e) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #3 nuraft::delayed_task_scheduler::cancel(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/delayed_task_scheduler.hxx:36:9 (clickhouse+0x1ceff126) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #4 nuraft::raft_server::cancel_task(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:367:17 (clickhouse+0x1ceff126) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #5 nuraft::raft_server::restart_election_timer() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:177:9 (clickhouse+0x1ceff126) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #6 nuraft::raft_server::handle_append_entries(nuraft::req_msg&) build_docker/./contrib/NuRaft/src/handle_append_entries.cxx:855:9 (clickhouse+0x1cf0890d) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #7 nuraft::raft_server::process_req(nuraft::req_msg&, nuraft::raft_server::req_ext_params const&) build_docker/./contrib/NuRaft/src/raft_server.cxx:706:16 (clickhouse+0x1cebe4d3) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #8 nuraft::raft_server_handler::process_req(nuraft::raft_server*, nuraft::req_msg&, nuraft::raft_server::req_ext_params const&) build_docker/./contrib/NuRaft/include/libnuraft/raft_server_handler.hxx:42:21 (clickhouse+0x1ce99615) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #9 nuraft::rpc_session::read_complete(std::__1::shared_ptr, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:591:30 (clickhouse+0x1ce99615) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #10 nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)::operator()(boost::system::error_code const&, unsigned long) const build_docker/./contrib/NuRaft/src/asio_service.cxx:395:23 (clickhouse+0x1ce91655) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #11 boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>::operator()(boost::system::error_code, unsigned long, int) build_docker/./contrib/boost/boost/asio/impl/read.hpp:397:9 (clickhouse+0x1ceb1c8b) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #12 boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>::operator()() build_docker/./contrib/boost/boost/asio/detail/bind_handler.hpp:289:5 (clickhouse+0x1ceb1c8b) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #13 void boost::asio::asio_handler_invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, ...) build_docker/./contrib/boost/boost/asio/handler_invoke_hook.hpp:88:3 (clickhouse+0x1ceb17f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #14 void boost_asio_handler_invoke_helpers::invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)&) build_docker/./contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x1ceb17f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #15 void boost::asio::detail::asio_handler_invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>*) build_docker/./contrib/boost/boost/asio/impl/read.hpp:465:5 (clickhouse+0x1ceb17f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #16 void boost_asio_handler_invoke_helpers::invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>&) build_docker/./contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x1ceb17f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #17 void boost::asio::detail::handler_work, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor, void>::complete, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>&) build_docker/./contrib/boost/boost/asio/detail/handler_work.hpp:524:7 (clickhouse+0x1ceb17f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #18 boost::asio::detail::reactive_socket_recv_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/reactive_socket_recv_op.hpp:152:9 (clickhouse+0x1ceb17f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #19 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x1ce469f2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #20 boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/impl/epoll_reactor.ipp:826:11 (clickhouse+0x1ce469f2) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #21 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x1ce29f8e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #22 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) build_docker/./contrib/boost/boost/asio/detail/impl/scheduler.ipp:493:12 (clickhouse+0x1ce29f8e) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #23 boost::asio::detail::scheduler::run(boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/impl/scheduler.ipp:210:10 (clickhouse+0x1ce296e4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #24 boost::asio::io_context::run() build_docker/./contrib/boost/boost/asio/impl/io_context.ipp:64:24 (clickhouse+0x1ce20d1f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #25 nuraft::asio_service_impl::worker_entry() build_docker/./contrib/NuRaft/src/asio_service.cxx:1807:21 (clickhouse+0x1ce20d1f) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #26 decltype(*std::declval().*std::declval()()) std::__1::__invoke[abi:v15000](void (nuraft::asio_service_impl::*&)(), nuraft::asio_service_impl*&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:359:23 (clickhouse+0x1ce347fb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #27 std::__1::__bind_return, std::__1::tuple<>, __is_valid_bind_return, std::__1::tuple<>>::value>::type std::__1::__apply_functor[abi:v15000], 0ul, std::__1::tuple<>>(void (nuraft::asio_service_impl::*&)(), std::__1::tuple&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/bind.h:263:12 (clickhouse+0x1ce347fb) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #28 std::__1::__bind_return, std::__1::tuple<>, __is_valid_bind_return, std::__1::tuple<>>::value>::type std::__1::__bind::operator()[abi:v15000]<>() build_docker/./contrib/llvm-project/libcxx/include/__functional/bind.h:295:20 (clickhouse+0x1ce347fb) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #29 decltype(std::declval>()()) std::__1::__invoke[abi:v15000]>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1ce347fb) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #30 void std::__1::__thread_execute[abi:v15000]>, std::__1::__bind>(std::__1::tuple>, std::__1::__bind>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0x1ce347fb) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #31 void* std::__1::__thread_proxy[abi:v15000]>, std::__1::__bind>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0x1ce347fb) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: Previous write of size 1 at 0x7b2000015c10 by main thread (mutexes: write M1): (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #0 boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:163:35 (clickhouse+0x1ce22a7f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #1 boost::asio::basic_waitable_timer, boost::asio::any_io_executor>::cancel() build_docker/./contrib/boost/boost/asio/basic_waitable_timer.hpp:415:41 (clickhouse+0x1ce22a7f) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #2 nuraft::asio_service::cancel_impl(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/asio_service.cxx:1925:71 (clickhouse+0x1ce22a7f) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #3 nuraft::delayed_task_scheduler::cancel(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/delayed_task_scheduler.hxx:36:9 (clickhouse+0x1cf006e9) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #4 nuraft::raft_server::cancel_task(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:367:17 (clickhouse+0x1cf006e9) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #5 nuraft::raft_server::cancel_schedulers() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:336:9 (clickhouse+0x1cf006e9) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #6 nuraft::raft_server::shutdown() build_docker/./contrib/NuRaft/src/raft_server.cxx:470:5 (clickhouse+0x1cebc41f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #7 DB::KeeperServer::shutdownRaftServer() build_docker/./src/Coordination/KeeperServer.cpp:543:20 (clickhouse+0x19e6a740) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #8 DB::KeeperServer::shutdown() build_docker/./src/Coordination/KeeperServer.cpp:579:5 (clickhouse+0x19e6b3f9) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #9 DB::KeeperDispatcher::shutdown() build_docker/./src/Coordination/KeeperDispatcher.cpp:598:21 (clickhouse+0x19e44dba) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #10 DB::Context::shutdownKeeperDispatcher() const build_docker/./src/Interpreters/Context.cpp:3467:36 (clickhouse+0x16829315) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #11 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1::operator()() const build_docker/./programs/server/Server.cpp:755:5 (clickhouse+0xf1b676c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #12 BasicScopeGuard, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1>::invoke() build_docker/./base/base/../base/scope_guard.h:101:9 (clickhouse+0xf1b676c) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #13 BasicScopeGuard, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1>::~BasicScopeGuard() build_docker/./base/base/../base/scope_guard.h:50:26 (clickhouse+0xf1b676c) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #14 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2167:1 (clickhouse+0xf1a541b) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #15 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #16 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #17 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #18 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #19 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: Location is heap block of size 120 at 0x7b2000015c00 allocated by main thread: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #0 operator new(unsigned long) (clickhouse+0x70a36f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #1 nuraft::asio_service::schedule(std::__1::shared_ptr&, int) build_docker/./contrib/NuRaft/src/asio_service.cxx:1907:33 (clickhouse+0x1ce22613) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #2 nuraft::raft_server::schedule_task(std::__1::shared_ptr&, int) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:361:21 (clickhouse+0x1ceff41c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #3 nuraft::raft_server::restart_election_timer() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:187:5 (clickhouse+0x1ceff41c) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #4 nuraft::raft_server::start_server(bool) build_docker/./contrib/NuRaft/src/raft_server.cxx:306:9 (clickhouse+0x1cebabff) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #5 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:498:20 (clickhouse+0x19e67d5c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #6 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #7 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #8 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #9 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #10 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #11 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #12 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #13 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #14 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: Mutex M0 (0x7b6c000049c8) created at: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #0 pthread_mutex_init (clickhouse+0x701ee90) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #1 std::__1::__libcpp_recursive_mutex_init[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:269:10 (clickhouse+0x202fba84) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #2 std::__1::recursive_mutex::recursive_mutex() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:61:14 (clickhouse+0x202fba84) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #3 nuraft::raft_server::raft_server(nuraft::context*, nuraft::raft_server::init_options const&) build_docker/./contrib/NuRaft/src/raft_server.cxx:50:14 (clickhouse+0x1ceb5eec) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #4 DB::KeeperServer::KeeperRaftServer::KeeperRaftServer(nuraft::context*, nuraft::raft_server::init_options const&) build_docker/./src/Coordination/KeeperServer.cpp:279:32 (clickhouse+0x19e82f10) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #5 DB::KeeperServer::KeeperRaftServer* std::__1::construct_at[abi:v15000](DB::KeeperServer::KeeperRaftServer*, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x19e82f10) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #6 void std::__1::allocator_traits>::construct[abi:v15000](std::__1::allocator&, DB::KeeperServer::KeeperRaftServer*, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x19e82f10) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #7 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000](std::__1::allocator, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x19e82f10) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #8 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], nuraft::context*&, nuraft::raft_server::init_options&, void>(std::__1::allocator const&, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x19e82f10) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #9 std::__1::shared_ptr std::__1::make_shared[abi:v15000](nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19e67a43) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #10 std::__1::shared_ptr nuraft::cs_new(nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x19e67a43) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #11 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:487:21 (clickhouse+0x19e67a43) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #12 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #13 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #14 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #15 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #17 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #20 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: Mutex M1 (0x7b9800002be8) created at: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #0 pthread_mutex_lock (clickhouse+0x701f1ab) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #1 std::__1::__libcpp_mutex_lock[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:304:10 (clickhouse+0x202fb979) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #2 std::__1::mutex::lock() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:38:14 (clickhouse+0x202fb979) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #3 std::__1::lock_guard::lock_guard[abi:v15000](std::__1::mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:94:27 (clickhouse+0x16829417) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #4 DB::Context::updateKeeperConfiguration(Poco::Util::AbstractConfiguration const&) build_docker/./src/Interpreters/Context.cpp:3477:21 (clickhouse+0x16829417) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #5 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6::operator()(Poco::AutoPtr, bool) const build_docker/./programs/server/Server.cpp:1457:33 (clickhouse+0xf1c0c06) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #6 decltype(std::declval, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&>()(std::declval>(), std::declval())) std::__1::__invoke[abi:v15000], std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr, bool>(DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf1bef49) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #7 void std::__1::__invoke_void_return_wrapper::__call, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr, bool>(DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf1bef49) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #8 std::__1::__function::__default_alloc_func, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, void (Poco::AutoPtr, bool)>::operator()[abi:v15000](Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf1bef49) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #9 void std::__1::__function::__policy_invoker, bool)>::__call_impl, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, void (Poco::AutoPtr, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf1bef49) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #10 std::__1::__function::__policy_func, bool)>::operator()[abi:v15000](Poco::AutoPtr&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fae13e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #11 std::__1::function, bool)>::operator()(Poco::AutoPtr, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fae13e) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #12 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fae13e) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #13 DB::ConfigReloader::ConfigReloader(std::__1::basic_string_view>, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&, std::__1::basic_string, std::__1::allocator> const&, zkutil::ZooKeeperNodeCache&&, std::__1::shared_ptr const&, std::__1::function, bool)>&&, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:32:9 (clickhouse+0x19facb5f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #14 std::__1::__unique_if::__unique_single std::__1::make_unique[abi:v15000], std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::basic_string, std::__1::allocator>, zkutil::ZooKeeperNodeCache, std::__1::shared_ptr&, DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, bool>(std::__1::basic_string, std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::basic_string, std::__1::allocator>&&, zkutil::ZooKeeperNodeCache&&, std::__1::shared_ptr&, DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0xf19f4b8) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #15 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1329:33 (clickhouse+0xf19f4b8) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #17 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #20 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: Thread T582 (tid=1457, running) created by main thread at: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #0 pthread_create (clickhouse+0x701d4ab) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #1 std::__1::__libcpp_thread_create[abi:v15000](unsigned long*, void* (*)(void*), void*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:376:10 (clickhouse+0x1ce346de) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #2 std::__1::thread::thread, void>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/thread:311:16 (clickhouse+0x1ce346de) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #3 std::__1::thread* std::__1::construct_at[abi:v15000], std::__1::thread*>(std::__1::thread*, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x1ce1ef38) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #4 void std::__1::allocator_traits>::construct[abi:v15000], void, void>(std::__1::allocator&, std::__1::thread*, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x1ce1ef38) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #5 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000]>(std::__1::allocator, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x1ce1ef38) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #6 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], std::__1::__bind, void>(std::__1::allocator const&, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x1ce1ef38) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #7 std::__1::shared_ptr std::__1::make_shared[abi:v15000], void>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x1ce1ef38) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #8 std::__1::shared_ptr nuraft::cs_new>(std::__1::__bind&&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x1ce1ef38) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #9 nuraft::asio_service_impl::asio_service_impl(nuraft::asio_service_options const&, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:1768:13 (clickhouse+0x1ce1ef38) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #10 nuraft::asio_service::asio_service(nuraft::asio_service_options const&, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:1897:17 (clickhouse+0x1ce22195) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #11 nuraft::asio_service* std::__1::construct_at[abi:v15000]&, nuraft::asio_service*>(nuraft::asio_service*, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x19e673f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #12 void std::__1::allocator_traits>::construct[abi:v15000]&, void, void>(std::__1::allocator&, nuraft::asio_service*, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x19e673f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #13 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000]&>(std::__1::allocator, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x19e673f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #14 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], nuraft::asio_service_options&, std::__1::shared_ptr&, void>(std::__1::allocator const&, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x19e673f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #15 std::__1::shared_ptr std::__1::make_shared[abi:v15000]&, void>(nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19e673f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #16 std::__1::shared_ptr nuraft::cs_new&>(nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x19e673f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #17 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:455:20 (clickhouse+0x19e673f7) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #18 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #19 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #20 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #21 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #22 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #23 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #24 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #25 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: #26 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout: (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout:SUMMARY: ThreadSanitizer: data race build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:153:15 in boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout:================== (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout:ThreadSanitizer: reported 1 warnings (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout:Logging test to /var/log/clickhouse-server/clickhouse-server.log (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stdout:Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:117, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : grep result ================== WARNING: ThreadSanitizer: data race (pid=849) Read of size 1 at 0x7b2000015c10 by thread T582 (mutexes: write M0): #0 boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:153:15 (clickhouse+0x1ce22a3e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 boost::asio::basic_waitable_timer, boost::asio::any_io_executor>::cancel() build_docker/./contrib/boost/boost/asio/basic_waitable_timer.hpp:415:41 (clickhouse+0x1ce22a3e) #2 nuraft::asio_service::cancel_impl(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/asio_service.cxx:1925:71 (clickhouse+0x1ce22a3e) #3 nuraft::delayed_task_scheduler::cancel(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/delayed_task_scheduler.hxx:36:9 (clickhouse+0x1ceff126) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 nuraft::raft_server::cancel_task(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:367:17 (clickhouse+0x1ceff126) #5 nuraft::raft_server::restart_election_timer() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:177:9 (clickhouse+0x1ceff126) #6 nuraft::raft_server::handle_append_entries(nuraft::req_msg&) build_docker/./contrib/NuRaft/src/handle_append_entries.cxx:855:9 (clickhouse+0x1cf0890d) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #7 nuraft::raft_server::process_req(nuraft::req_msg&, nuraft::raft_server::req_ext_params const&) build_docker/./contrib/NuRaft/src/raft_server.cxx:706:16 (clickhouse+0x1cebe4d3) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #8 nuraft::raft_server_handler::process_req(nuraft::raft_server*, nuraft::req_msg&, nuraft::raft_server::req_ext_params const&) build_docker/./contrib/NuRaft/include/libnuraft/raft_server_handler.hxx:42:21 (clickhouse+0x1ce99615) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #9 nuraft::rpc_session::read_complete(std::__1::shared_ptr, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:591:30 (clickhouse+0x1ce99615) #10 nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)::operator()(boost::system::error_code const&, unsigned long) const build_docker/./contrib/NuRaft/src/asio_service.cxx:395:23 (clickhouse+0x1ce91655) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>::operator()(boost::system::error_code, unsigned long, int) build_docker/./contrib/boost/boost/asio/impl/read.hpp:397:9 (clickhouse+0x1ceb1c8b) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #12 boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>::operator()() build_docker/./contrib/boost/boost/asio/detail/bind_handler.hpp:289:5 (clickhouse+0x1ceb1c8b) #13 void boost::asio::asio_handler_invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, ...) build_docker/./contrib/boost/boost/asio/handler_invoke_hook.hpp:88:3 (clickhouse+0x1ceb17f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #14 void boost_asio_handler_invoke_helpers::invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)&) build_docker/./contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x1ceb17f7) #15 void boost::asio::detail::asio_handler_invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>*) build_docker/./contrib/boost/boost/asio/impl/read.hpp:465:5 (clickhouse+0x1ceb17f7) #16 void boost_asio_handler_invoke_helpers::invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>&) build_docker/./contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x1ceb17f7) #17 void boost::asio::detail::handler_work, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor, void>::complete, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>&) build_docker/./contrib/boost/boost/asio/detail/handler_work.hpp:524:7 (clickhouse+0x1ceb17f7) #18 boost::asio::detail::reactive_socket_recv_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/reactive_socket_recv_op.hpp:152:9 (clickhouse+0x1ceb17f7) #19 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x1ce469f2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #20 boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/impl/epoll_reactor.ipp:826:11 (clickhouse+0x1ce469f2) #21 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x1ce29f8e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #22 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) build_docker/./contrib/boost/boost/asio/detail/impl/scheduler.ipp:493:12 (clickhouse+0x1ce29f8e) #23 boost::asio::detail::scheduler::run(boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/impl/scheduler.ipp:210:10 (clickhouse+0x1ce296e4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #24 boost::asio::io_context::run() build_docker/./contrib/boost/boost/asio/impl/io_context.ipp:64:24 (clickhouse+0x1ce20d1f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #25 nuraft::asio_service_impl::worker_entry() build_docker/./contrib/NuRaft/src/asio_service.cxx:1807:21 (clickhouse+0x1ce20d1f) #26 decltype(*std::declval().*std::declval()()) std::__1::__invoke[abi:v15000](void (nuraft::asio_service_impl::*&)(), nuraft::asio_service_impl*&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:359:23 (clickhouse+0x1ce347fb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #27 std::__1::__bind_return, std::__1::tuple<>, __is_valid_bind_return, std::__1::tuple<>>::value>::type std::__1::__apply_functor[abi:v15000], 0ul, std::__1::tuple<>>(void (nuraft::asio_service_impl::*&)(), std::__1::tuple&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/bind.h:263:12 (clickhouse+0x1ce347fb) #28 std::__1::__bind_return, std::__1::tuple<>, __is_valid_bind_return, std::__1::tuple<>>::value>::type std::__1::__bind::operator()[abi:v15000]<>() build_docker/./contrib/llvm-project/libcxx/include/__functional/bind.h:295:20 (clickhouse+0x1ce347fb) #29 decltype(std::declval>()()) std::__1::__invoke[abi:v15000]>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1ce347fb) #30 void std::__1::__thread_execute[abi:v15000]>, std::__1::__bind>(std::__1::tuple>, std::__1::__bind>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0x1ce347fb) #31 void* std::__1::__thread_proxy[abi:v15000]>, std::__1::__bind>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0x1ce347fb) Previous write of size 1 at 0x7b2000015c10 by main thread (mutexes: write M1): #0 boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:163:35 (clickhouse+0x1ce22a7f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 boost::asio::basic_waitable_timer, boost::asio::any_io_executor>::cancel() build_docker/./contrib/boost/boost/asio/basic_waitable_timer.hpp:415:41 (clickhouse+0x1ce22a7f) #2 nuraft::asio_service::cancel_impl(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/asio_service.cxx:1925:71 (clickhouse+0x1ce22a7f) #3 nuraft::delayed_task_scheduler::cancel(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/delayed_task_scheduler.hxx:36:9 (clickhouse+0x1cf006e9) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 nuraft::raft_server::cancel_task(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:367:17 (clickhouse+0x1cf006e9) #5 nuraft::raft_server::cancel_schedulers() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:336:9 (clickhouse+0x1cf006e9) #6 nuraft::raft_server::shutdown() build_docker/./contrib/NuRaft/src/raft_server.cxx:470:5 (clickhouse+0x1cebc41f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #7 DB::KeeperServer::shutdownRaftServer() build_docker/./src/Coordination/KeeperServer.cpp:543:20 (clickhouse+0x19e6a740) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #8 DB::KeeperServer::shutdown() build_docker/./src/Coordination/KeeperServer.cpp:579:5 (clickhouse+0x19e6b3f9) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #9 DB::KeeperDispatcher::shutdown() build_docker/./src/Coordination/KeeperDispatcher.cpp:598:21 (clickhouse+0x19e44dba) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #10 DB::Context::shutdownKeeperDispatcher() const build_docker/./src/Interpreters/Context.cpp:3467:36 (clickhouse+0x16829315) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1::operator()() const build_docker/./programs/server/Server.cpp:755:5 (clickhouse+0xf1b676c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #12 BasicScopeGuard, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1>::invoke() build_docker/./base/base/../base/scope_guard.h:101:9 (clickhouse+0xf1b676c) #13 BasicScopeGuard, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1>::~BasicScopeGuard() build_docker/./base/base/../base/scope_guard.h:50:26 (clickhouse+0xf1b676c) #14 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2167:1 (clickhouse+0xf1a541b) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #15 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #16 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #17 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #18 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #19 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) Location is heap block of size 120 at 0x7b2000015c00 allocated by main thread: #0 operator new(unsigned long) (clickhouse+0x70a36f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 nuraft::asio_service::schedule(std::__1::shared_ptr&, int) build_docker/./contrib/NuRaft/src/asio_service.cxx:1907:33 (clickhouse+0x1ce22613) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #2 nuraft::raft_server::schedule_task(std::__1::shared_ptr&, int) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:361:21 (clickhouse+0x1ceff41c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #3 nuraft::raft_server::restart_election_timer() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:187:5 (clickhouse+0x1ceff41c) #4 nuraft::raft_server::start_server(bool) build_docker/./contrib/NuRaft/src/raft_server.cxx:306:9 (clickhouse+0x1cebabff) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #5 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:498:20 (clickhouse+0x19e67d5c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #6 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #7 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #8 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #9 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #10 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #12 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #13 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #14 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) Mutex M0 (0x7b6c000049c8) created at: #0 pthread_mutex_init (clickhouse+0x701ee90) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 std::__1::__libcpp_recursive_mutex_init[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:269:10 (clickhouse+0x202fba84) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #2 std::__1::recursive_mutex::recursive_mutex() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:61:14 (clickhouse+0x202fba84) #3 nuraft::raft_server::raft_server(nuraft::context*, nuraft::raft_server::init_options const&) build_docker/./contrib/NuRaft/src/raft_server.cxx:50:14 (clickhouse+0x1ceb5eec) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 DB::KeeperServer::KeeperRaftServer::KeeperRaftServer(nuraft::context*, nuraft::raft_server::init_options const&) build_docker/./src/Coordination/KeeperServer.cpp:279:32 (clickhouse+0x19e82f10) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #5 DB::KeeperServer::KeeperRaftServer* std::__1::construct_at[abi:v15000](DB::KeeperServer::KeeperRaftServer*, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x19e82f10) #6 void std::__1::allocator_traits>::construct[abi:v15000](std::__1::allocator&, DB::KeeperServer::KeeperRaftServer*, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x19e82f10) #7 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000](std::__1::allocator, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x19e82f10) #8 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], nuraft::context*&, nuraft::raft_server::init_options&, void>(std::__1::allocator const&, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x19e82f10) #9 std::__1::shared_ptr std::__1::make_shared[abi:v15000](nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19e67a43) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #10 std::__1::shared_ptr nuraft::cs_new(nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x19e67a43) #11 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:487:21 (clickhouse+0x19e67a43) #12 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #13 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #14 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #15 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #17 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #20 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) Mutex M1 (0x7b9800002be8) created at: #0 pthread_mutex_lock (clickhouse+0x701f1ab) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 std::__1::__libcpp_mutex_lock[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:304:10 (clickhouse+0x202fb979) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #2 std::__1::mutex::lock() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:38:14 (clickhouse+0x202fb979) #3 std::__1::lock_guard::lock_guard[abi:v15000](std::__1::mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:94:27 (clickhouse+0x16829417) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 DB::Context::updateKeeperConfiguration(Poco::Util::AbstractConfiguration const&) build_docker/./src/Interpreters/Context.cpp:3477:21 (clickhouse+0x16829417) #5 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6::operator()(Poco::AutoPtr, bool) const build_docker/./programs/server/Server.cpp:1457:33 (clickhouse+0xf1c0c06) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #6 decltype(std::declval, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&>()(std::declval>(), std::declval())) std::__1::__invoke[abi:v15000], std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr, bool>(DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf1bef49) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #7 void std::__1::__invoke_void_return_wrapper::__call, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr, bool>(DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf1bef49) #8 std::__1::__function::__default_alloc_func, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, void (Poco::AutoPtr, bool)>::operator()[abi:v15000](Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf1bef49) #9 void std::__1::__function::__policy_invoker, bool)>::__call_impl, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, void (Poco::AutoPtr, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf1bef49) #10 std::__1::__function::__policy_func, bool)>::operator()[abi:v15000](Poco::AutoPtr&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fae13e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 std::__1::function, bool)>::operator()(Poco::AutoPtr, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fae13e) #12 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fae13e) #13 DB::ConfigReloader::ConfigReloader(std::__1::basic_string_view>, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&, std::__1::basic_string, std::__1::allocator> const&, zkutil::ZooKeeperNodeCache&&, std::__1::shared_ptr const&, std::__1::function, bool)>&&, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:32:9 (clickhouse+0x19facb5f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #14 std::__1::__unique_if::__unique_single std::__1::make_unique[abi:v15000], std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::basic_string, std::__1::allocator>, zkutil::ZooKeeperNodeCache, std::__1::shared_ptr&, DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, bool>(std::__1::basic_string, std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::basic_string, std::__1::allocator>&&, zkutil::ZooKeeperNodeCache&&, std::__1::shared_ptr&, DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0xf19f4b8) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #15 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1329:33 (clickhouse+0xf19f4b8) #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #17 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #20 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) Thread T582 (tid=1457, running) created by main thread at: #0 pthread_create (clickhouse+0x701d4ab) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 std::__1::__libcpp_thread_create[abi:v15000](unsigned long*, void* (*)(void*), void*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:376:10 (clickhouse+0x1ce346de) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #2 std::__1::thread::thread, void>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/thread:311:16 (clickhouse+0x1ce346de) #3 std::__1::thread* std::__1::construct_at[abi:v15000], std::__1::thread*>(std::__1::thread*, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x1ce1ef38) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 void std::__1::allocator_traits>::construct[abi:v15000], void, void>(std::__1::allocator&, std::__1::thread*, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x1ce1ef38) #5 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000]>(std::__1::allocator, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x1ce1ef38) #6 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], std::__1::__bind, void>(std::__1::allocator const&, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x1ce1ef38) #7 std::__1::shared_ptr std::__1::make_shared[abi:v15000], void>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x1ce1ef38) #8 std::__1::shared_ptr nuraft::cs_new>(std::__1::__bind&&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x1ce1ef38) #9 nuraft::asio_service_impl::asio_service_impl(nuraft::asio_service_options const&, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:1768:13 (clickhouse+0x1ce1ef38) #10 nuraft::asio_service::asio_service(nuraft::asio_service_options const&, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:1897:17 (clickhouse+0x1ce22195) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 nuraft::asio_service* std::__1::construct_at[abi:v15000]&, nuraft::asio_service*>(nuraft::asio_service*, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x19e673f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #12 void std::__1::allocator_traits>::construct[abi:v15000]&, void, void>(std::__1::allocator&, nuraft::asio_service*, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x19e673f7) #13 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000]&>(std::__1::allocator, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x19e673f7) #14 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], nuraft::asio_service_options&, std::__1::shared_ptr&, void>(std::__1::allocator const&, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x19e673f7) #15 std::__1::shared_ptr std::__1::make_shared[abi:v15000]&, void>(nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19e673f7) #16 std::__1::shared_ptr nuraft::cs_new&>(nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x19e673f7) #17 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:455:20 (clickhouse+0x19e673f7) #18 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #19 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #20 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #21 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #22 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #23 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #24 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #25 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #26 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) SUMMARY: ThreadSanitizer: data race build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:153:15 in boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) ================== ThreadSanitizer: reported 1 warnings Logging test to /var/log/clickhouse-server/clickhouse-server.log Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:3963, grep_in_log) 2025-03-05 20:32:05 [ 554 ] ERROR : Sanitizer in instance node1 log ================== WARNING: ThreadSanitizer: data race (pid=849) Read of size 1 at 0x7b2000015c10 by thread T582 (mutexes: write M0): #0 boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:153:15 (clickhouse+0x1ce22a3e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 boost::asio::basic_waitable_timer, boost::asio::any_io_executor>::cancel() build_docker/./contrib/boost/boost/asio/basic_waitable_timer.hpp:415:41 (clickhouse+0x1ce22a3e) #2 nuraft::asio_service::cancel_impl(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/asio_service.cxx:1925:71 (clickhouse+0x1ce22a3e) #3 nuraft::delayed_task_scheduler::cancel(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/delayed_task_scheduler.hxx:36:9 (clickhouse+0x1ceff126) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 nuraft::raft_server::cancel_task(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:367:17 (clickhouse+0x1ceff126) #5 nuraft::raft_server::restart_election_timer() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:177:9 (clickhouse+0x1ceff126) #6 nuraft::raft_server::handle_append_entries(nuraft::req_msg&) build_docker/./contrib/NuRaft/src/handle_append_entries.cxx:855:9 (clickhouse+0x1cf0890d) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #7 nuraft::raft_server::process_req(nuraft::req_msg&, nuraft::raft_server::req_ext_params const&) build_docker/./contrib/NuRaft/src/raft_server.cxx:706:16 (clickhouse+0x1cebe4d3) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #8 nuraft::raft_server_handler::process_req(nuraft::raft_server*, nuraft::req_msg&, nuraft::raft_server::req_ext_params const&) build_docker/./contrib/NuRaft/include/libnuraft/raft_server_handler.hxx:42:21 (clickhouse+0x1ce99615) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #9 nuraft::rpc_session::read_complete(std::__1::shared_ptr, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:591:30 (clickhouse+0x1ce99615) #10 nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)::operator()(boost::system::error_code const&, unsigned long) const build_docker/./contrib/NuRaft/src/asio_service.cxx:395:23 (clickhouse+0x1ce91655) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>::operator()(boost::system::error_code, unsigned long, int) build_docker/./contrib/boost/boost/asio/impl/read.hpp:397:9 (clickhouse+0x1ceb1c8b) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #12 boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>::operator()() build_docker/./contrib/boost/boost/asio/detail/bind_handler.hpp:289:5 (clickhouse+0x1ceb1c8b) #13 void boost::asio::asio_handler_invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, ...) build_docker/./contrib/boost/boost/asio/handler_invoke_hook.hpp:88:3 (clickhouse+0x1ceb17f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #14 void boost_asio_handler_invoke_helpers::invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)&) build_docker/./contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x1ceb17f7) #15 void boost::asio::detail::asio_handler_invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>*) build_docker/./contrib/boost/boost/asio/impl/read.hpp:465:5 (clickhouse+0x1ceb17f7) #16 void boost_asio_handler_invoke_helpers::invoke, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>&) build_docker/./contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x1ceb17f7) #17 void boost::asio::detail::handler_work, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor, void>::complete, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>>(boost::asio::detail::binder2, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>&) build_docker/./contrib/boost/boost/asio/detail/handler_work.hpp:524:7 (clickhouse+0x1ceb17f7) #18 boost::asio::detail::reactive_socket_recv_op, boost::asio::mutable_buffers_1, boost::asio::mutable_buffer const*, boost::asio::detail::transfer_all_t, nuraft::rpc_session::start(std::__1::shared_ptr)::'lambda'(boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/reactive_socket_recv_op.hpp:152:9 (clickhouse+0x1ceb17f7) #19 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x1ce469f2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #20 boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/impl/epoll_reactor.ipp:826:11 (clickhouse+0x1ce469f2) #21 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/./contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x1ce29f8e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #22 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) build_docker/./contrib/boost/boost/asio/detail/impl/scheduler.ipp:493:12 (clickhouse+0x1ce29f8e) #23 boost::asio::detail::scheduler::run(boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/impl/scheduler.ipp:210:10 (clickhouse+0x1ce296e4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #24 boost::asio::io_context::run() build_docker/./contrib/boost/boost/asio/impl/io_context.ipp:64:24 (clickhouse+0x1ce20d1f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #25 nuraft::asio_service_impl::worker_entry() build_docker/./contrib/NuRaft/src/asio_service.cxx:1807:21 (clickhouse+0x1ce20d1f) #26 decltype(*std::declval().*std::declval()()) std::__1::__invoke[abi:v15000](void (nuraft::asio_service_impl::*&)(), nuraft::asio_service_impl*&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:359:23 (clickhouse+0x1ce347fb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #27 std::__1::__bind_return, std::__1::tuple<>, __is_valid_bind_return, std::__1::tuple<>>::value>::type std::__1::__apply_functor[abi:v15000], 0ul, std::__1::tuple<>>(void (nuraft::asio_service_impl::*&)(), std::__1::tuple&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/bind.h:263:12 (clickhouse+0x1ce347fb) #28 std::__1::__bind_return, std::__1::tuple<>, __is_valid_bind_return, std::__1::tuple<>>::value>::type std::__1::__bind::operator()[abi:v15000]<>() build_docker/./contrib/llvm-project/libcxx/include/__functional/bind.h:295:20 (clickhouse+0x1ce347fb) #29 decltype(std::declval>()()) std::__1::__invoke[abi:v15000]>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1ce347fb) #30 void std::__1::__thread_execute[abi:v15000]>, std::__1::__bind>(std::__1::tuple>, std::__1::__bind>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0x1ce347fb) #31 void* std::__1::__thread_proxy[abi:v15000]>, std::__1::__bind>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0x1ce347fb) Previous write of size 1 at 0x7b2000015c10 by main thread (mutexes: write M1): #0 boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:163:35 (clickhouse+0x1ce22a7f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 boost::asio::basic_waitable_timer, boost::asio::any_io_executor>::cancel() build_docker/./contrib/boost/boost/asio/basic_waitable_timer.hpp:415:41 (clickhouse+0x1ce22a7f) #2 nuraft::asio_service::cancel_impl(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/asio_service.cxx:1925:71 (clickhouse+0x1ce22a7f) #3 nuraft::delayed_task_scheduler::cancel(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/delayed_task_scheduler.hxx:36:9 (clickhouse+0x1cf006e9) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 nuraft::raft_server::cancel_task(std::__1::shared_ptr&) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:367:17 (clickhouse+0x1cf006e9) #5 nuraft::raft_server::cancel_schedulers() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:336:9 (clickhouse+0x1cf006e9) #6 nuraft::raft_server::shutdown() build_docker/./contrib/NuRaft/src/raft_server.cxx:470:5 (clickhouse+0x1cebc41f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #7 DB::KeeperServer::shutdownRaftServer() build_docker/./src/Coordination/KeeperServer.cpp:543:20 (clickhouse+0x19e6a740) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #8 DB::KeeperServer::shutdown() build_docker/./src/Coordination/KeeperServer.cpp:579:5 (clickhouse+0x19e6b3f9) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #9 DB::KeeperDispatcher::shutdown() build_docker/./src/Coordination/KeeperDispatcher.cpp:598:21 (clickhouse+0x19e44dba) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #10 DB::Context::shutdownKeeperDispatcher() const build_docker/./src/Interpreters/Context.cpp:3467:36 (clickhouse+0x16829315) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1::operator()() const build_docker/./programs/server/Server.cpp:755:5 (clickhouse+0xf1b676c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #12 BasicScopeGuard, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1>::invoke() build_docker/./base/base/../base/scope_guard.h:101:9 (clickhouse+0xf1b676c) #13 BasicScopeGuard, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_1>::~BasicScopeGuard() build_docker/./base/base/../base/scope_guard.h:50:26 (clickhouse+0xf1b676c) #14 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2167:1 (clickhouse+0xf1a541b) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #15 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #16 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #17 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #18 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #19 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) Location is heap block of size 120 at 0x7b2000015c00 allocated by main thread: #0 operator new(unsigned long) (clickhouse+0x70a36f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 nuraft::asio_service::schedule(std::__1::shared_ptr&, int) build_docker/./contrib/NuRaft/src/asio_service.cxx:1907:33 (clickhouse+0x1ce22613) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #2 nuraft::raft_server::schedule_task(std::__1::shared_ptr&, int) build_docker/./contrib/NuRaft/src/handle_timeout.cxx:361:21 (clickhouse+0x1ceff41c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #3 nuraft::raft_server::restart_election_timer() build_docker/./contrib/NuRaft/src/handle_timeout.cxx:187:5 (clickhouse+0x1ceff41c) #4 nuraft::raft_server::start_server(bool) build_docker/./contrib/NuRaft/src/raft_server.cxx:306:9 (clickhouse+0x1cebabff) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #5 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:498:20 (clickhouse+0x19e67d5c) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #6 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #7 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #8 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #9 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #10 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #12 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #13 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #14 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) Mutex M0 (0x7b6c000049c8) created at: #0 pthread_mutex_init (clickhouse+0x701ee90) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 std::__1::__libcpp_recursive_mutex_init[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:269:10 (clickhouse+0x202fba84) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #2 std::__1::recursive_mutex::recursive_mutex() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:61:14 (clickhouse+0x202fba84) #3 nuraft::raft_server::raft_server(nuraft::context*, nuraft::raft_server::init_options const&) build_docker/./contrib/NuRaft/src/raft_server.cxx:50:14 (clickhouse+0x1ceb5eec) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 DB::KeeperServer::KeeperRaftServer::KeeperRaftServer(nuraft::context*, nuraft::raft_server::init_options const&) build_docker/./src/Coordination/KeeperServer.cpp:279:32 (clickhouse+0x19e82f10) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #5 DB::KeeperServer::KeeperRaftServer* std::__1::construct_at[abi:v15000](DB::KeeperServer::KeeperRaftServer*, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x19e82f10) #6 void std::__1::allocator_traits>::construct[abi:v15000](std::__1::allocator&, DB::KeeperServer::KeeperRaftServer*, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x19e82f10) #7 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000](std::__1::allocator, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x19e82f10) #8 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], nuraft::context*&, nuraft::raft_server::init_options&, void>(std::__1::allocator const&, nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x19e82f10) #9 std::__1::shared_ptr std::__1::make_shared[abi:v15000](nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19e67a43) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #10 std::__1::shared_ptr nuraft::cs_new(nuraft::context*&, nuraft::raft_server::init_options&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x19e67a43) #11 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:487:21 (clickhouse+0x19e67a43) #12 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #13 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #14 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #15 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #17 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #20 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) Mutex M1 (0x7b9800002be8) created at: #0 pthread_mutex_lock (clickhouse+0x701f1ab) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 std::__1::__libcpp_mutex_lock[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:304:10 (clickhouse+0x202fb979) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #2 std::__1::mutex::lock() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:38:14 (clickhouse+0x202fb979) #3 std::__1::lock_guard::lock_guard[abi:v15000](std::__1::mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:94:27 (clickhouse+0x16829417) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 DB::Context::updateKeeperConfiguration(Poco::Util::AbstractConfiguration const&) build_docker/./src/Interpreters/Context.cpp:3477:21 (clickhouse+0x16829417) #5 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6::operator()(Poco::AutoPtr, bool) const build_docker/./programs/server/Server.cpp:1457:33 (clickhouse+0xf1c0c06) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #6 decltype(std::declval, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&>()(std::declval>(), std::declval())) std::__1::__invoke[abi:v15000], std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr, bool>(DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf1bef49) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #7 void std::__1::__invoke_void_return_wrapper::__call, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr, bool>(DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&, Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf1bef49) #8 std::__1::__function::__default_alloc_func, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, void (Poco::AutoPtr, bool)>::operator()[abi:v15000](Poco::AutoPtr&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf1bef49) #9 void std::__1::__function::__policy_invoker, bool)>::__call_impl, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, void (Poco::AutoPtr, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf1bef49) #10 std::__1::__function::__policy_func, bool)>::operator()[abi:v15000](Poco::AutoPtr&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fae13e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 std::__1::function, bool)>::operator()(Poco::AutoPtr, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fae13e) #12 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fae13e) #13 DB::ConfigReloader::ConfigReloader(std::__1::basic_string_view>, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&, std::__1::basic_string, std::__1::allocator> const&, zkutil::ZooKeeperNodeCache&&, std::__1::shared_ptr const&, std::__1::function, bool)>&&, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:32:9 (clickhouse+0x19facb5f) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #14 std::__1::__unique_if::__unique_single std::__1::make_unique[abi:v15000], std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::basic_string, std::__1::allocator>, zkutil::ZooKeeperNodeCache, std::__1::shared_ptr&, DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6, bool>(std::__1::basic_string, std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::basic_string, std::__1::allocator>&&, zkutil::ZooKeeperNodeCache&&, std::__1::shared_ptr&, DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&)::$_6&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0xf19f4b8) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #15 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1329:33 (clickhouse+0xf19f4b8) #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #17 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #20 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) Thread T582 (tid=1457, running) created by main thread at: #0 pthread_create (clickhouse+0x701d4ab) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #1 std::__1::__libcpp_thread_create[abi:v15000](unsigned long*, void* (*)(void*), void*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:376:10 (clickhouse+0x1ce346de) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #2 std::__1::thread::thread, void>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/thread:311:16 (clickhouse+0x1ce346de) #3 std::__1::thread* std::__1::construct_at[abi:v15000], std::__1::thread*>(std::__1::thread*, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x1ce1ef38) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #4 void std::__1::allocator_traits>::construct[abi:v15000], void, void>(std::__1::allocator&, std::__1::thread*, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x1ce1ef38) #5 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000]>(std::__1::allocator, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x1ce1ef38) #6 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], std::__1::__bind, void>(std::__1::allocator const&, std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x1ce1ef38) #7 std::__1::shared_ptr std::__1::make_shared[abi:v15000], void>(std::__1::__bind&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x1ce1ef38) #8 std::__1::shared_ptr nuraft::cs_new>(std::__1::__bind&&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x1ce1ef38) #9 nuraft::asio_service_impl::asio_service_impl(nuraft::asio_service_options const&, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:1768:13 (clickhouse+0x1ce1ef38) #10 nuraft::asio_service::asio_service(nuraft::asio_service_options const&, std::__1::shared_ptr) build_docker/./contrib/NuRaft/src/asio_service.cxx:1897:17 (clickhouse+0x1ce22195) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #11 nuraft::asio_service* std::__1::construct_at[abi:v15000]&, nuraft::asio_service*>(nuraft::asio_service*, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x19e673f7) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #12 void std::__1::allocator_traits>::construct[abi:v15000]&, void, void>(std::__1::allocator&, nuraft::asio_service*, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x19e673f7) #13 std::__1::__shared_ptr_emplace>::__shared_ptr_emplace[abi:v15000]&>(std::__1::allocator, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x19e673f7) #14 std::__1::shared_ptr std::__1::allocate_shared[abi:v15000], nuraft::asio_service_options&, std::__1::shared_ptr&, void>(std::__1::allocator const&, nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x19e673f7) #15 std::__1::shared_ptr std::__1::make_shared[abi:v15000]&, void>(nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19e673f7) #16 std::__1::shared_ptr nuraft::cs_new&>(nuraft::asio_service_options&, std::__1::shared_ptr&) build_docker/./contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x19e673f7) #17 DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:455:20 (clickhouse+0x19e673f7) #18 DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:528:5 (clickhouse+0x19e6a3bc) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #19 DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (clickhouse+0x19e40ea2) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #20 DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (clickhouse+0x16828aeb) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #21 DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1611:25 (clickhouse+0xf19f712) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #22 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d01a71e) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #23 DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (clickhouse+0xf18e3e1) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #24 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d03a4b4) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #25 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (clickhouse+0xf18b135) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) #26 main build_docker/./programs/main.cpp:498:21 (clickhouse+0x70a5998) (BuildId: 2f0f4fe8dec66924dac7eb23ed924f473e1d5318) SUMMARY: ThreadSanitizer: data race build_docker/./contrib/boost/boost/asio/detail/deadline_timer_service.hpp:153:15 in boost::asio::detail::deadline_timer_service>>::cancel(boost::asio::detail::deadline_timer_service>>::implementation_type&, boost::system::error_code&) ================== ThreadSanitizer: reported 1 warnings Logging test to /var/log/clickhouse-server/clickhouse-server.log Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:3075, shutdown) 2025-03-05 20:32:05 [ 554 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/logs/stderr.log* || true'] (cluster.py:109, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/logs/stderr.log* || true'] (cluster.py:109, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/.env', '--project-name', 'roottestkeeperforcerecoverysinglenode', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/node3/docker-compose.yml', 'down', '--volumes'] (cluster.py:109, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stderr:Removing roottestkeeperforcerecoverysinglenode_node1_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stderr:Removing roottestkeeperforcerecoverysinglenode_node3_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stderr:Removing roottestkeeperforcerecoverysinglenode_node2_1 ... (cluster.py:119, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stderr:Removing roottestkeeperforcerecoverysinglenode_node1_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stderr:Removing roottestkeeperforcerecoverysinglenode_node2_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stderr:Removing roottestkeeperforcerecoverysinglenode_node3_1 ... done (cluster.py:119, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Stderr:Removing network roottestkeeperforcerecoverysinglenode_default (cluster.py:119, run_and_check) 2025-03-05 20:32:05 [ 554 ] DEBUG : Cleanup called (cluster.py:797, cleanup) 2025-03-05 20:32:05 [ 554 ] DEBUG : Docker networks for project roottestkeeperforcerecoverysinglenode are NETWORK ID NAME DRIVER SCOPE (cluster.py:776, print_all_docker_pieces) 2025-03-05 20:32:05 [ 554 ] DEBUG : Docker containers for project roottestkeeperforcerecoverysinglenode are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:784, print_all_docker_pieces) 2025-03-05 20:32:05 [ 554 ] DEBUG : Docker volumes for project roottestkeeperforcerecoverysinglenode are DRIVER VOLUME NAME (cluster.py:792, print_all_docker_pieces) 2025-03-05 20:32:05 [ 554 ] DEBUG : Command:docker container list --all --filter name='^/roottestkeeperforcerecoverysinglenode_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:109, run_and_check) 2025-03-05 20:32:06 [ 554 ] DEBUG : Unstopped containers: {} (cluster.py:811, cleanup) 2025-03-05 20:32:06 [ 554 ] DEBUG : No running containers for project: roottestkeeperforcerecoverysinglenode (cluster.py:825, cleanup) 2025-03-05 20:32:06 [ 554 ] DEBUG : Trying to prune unused networks... (cluster.py:831, cleanup) 2025-03-05 20:32:06 [ 554 ] DEBUG : Trying to prune unused images... (cluster.py:847, cleanup) 2025-03-05 20:32:06 [ 554 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:109, run_and_check) 2025-03-05 20:32:06 [ 554 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:117, run_and_check) 2025-03-05 20:32:06 [ 554 ] DEBUG : Images pruned (cluster.py:850, cleanup) 2025-03-05 20:32:06 [ 554 ] DEBUG : Trying to prune unused volumes... (cluster.py:856, cleanup) 2025-03-05 20:32:06 [ 554 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:109, run_and_check) 2025-03-05 20:32:06 [ 554 ] DEBUG : Stdout:6 (cluster.py:117, run_and_check) ============================== slowest durations =============================== 279.54s call test_kafka_bad_messages/test.py::test_bad_messages_parsing_stream 115.03s call test_filesystem_cache/test.py::test_force_filesystem_cache_on_merges 110.46s call test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 60.95s call test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 58.90s call test_keeper_mntr_pressure/test.py::test_aggressive_mntr 58.10s call test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch 40.08s call test_keeper_nodes_move/test.py::test_node_move 37.24s call test_mask_sensitive_info/test.py::test_table_functions 37.08s call test_keeper_reconfig_replace_leader_in_one_command/test.py::test_reconfig_replace_leader_in_one_command 33.89s call test_keeper_back_to_back/test.py::test_random_requests 33.72s setup test_kafka_bad_messages/test.py::test_bad_messages_parsing_exception 32.36s setup test_recovery_replica/test.py::test_choose_source_replica 30.86s setup test_jbod_balancer/test.py::test_jbod_balanced_merge 27.93s call test_keeper_back_to_back/test.py::test_concurrent_watches 26.26s call test_filesystem_cache/test.py::test_parallel_cache_loading_on_startup[node] 25.65s call test_mask_sensitive_info/test.py::test_create_table 23.86s setup test_intersecting_parts/test.py::test_intersect_parts_when_restart 23.69s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 22.94s teardown test_join_set_family_s3/test.py::test_set_s3 22.83s teardown test_log_family_s3/test.py::test_log_family_s3[TinyLog] 22.20s setup test_mask_sensitive_info/test.py::test_backup_table 21.16s call test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 20.90s setup test_part_uuid/test.py::test_part_uuid 20.81s setup test_keeper_client/test.py::test_base_commands 20.30s call test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 19.89s call test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size 19.87s call test_filesystem_cache/test.py::test_custom_cached_disk 19.32s teardown test_kafka_bad_messages/test.py::test_bad_messages_to_mv 19.08s setup test_keeper_back_to_back/test.py::test_concurrent_watches 18.79s setup test_modify_engine_on_restart/test_unusual_path.py::test_modify_engine_on_restart_with_unusual_path 18.06s call test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 17.99s call test_jbod_balancer/test.py::test_jbod_balanced_merge 17.98s setup test_keeper_four_word_command/test.py::test_cmd_clrs 17.97s setup test_log_family_s3/test.py::test_log_family_s3[Log] 17.40s setup test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 17.39s setup test_filesystem_cache/test.py::test_caches_with_the_same_configuration_2[node_caches_with_same_path] 17.24s setup test_multi_access_storage_role_management/test.py::test_quotas 16.84s setup test_merge_table_over_distributed/test.py::test_filtering 16.73s setup test_keeper_nodes_move/test.py::test_node_move 16.44s setup test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 16.36s setup test_join_set_family_s3/test.py::test_join_s3 16.23s call test_kafka_bad_messages/test.py::test_bad_messages_to_mv 15.69s setup test_keeper_session/test.py::test_session_close_shutdown 15.40s setup test_keeper_four_word_command/test_allow_list.py::test_allow_list 15.28s setup test_keeper_mntr_pressure/test.py::test_aggressive_mntr 14.59s setup test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 13.88s setup test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 13.71s setup test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 13.68s setup test_logs_level/test.py::test_check_client_logs_level 13.26s setup test_http_native/test.py::test_http_native_returns_timezone 13.12s setup test_read_temporary_tables_on_failure/test.py::test_different_versions 12.94s setup test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 12.86s call test_join_set_family_s3/test.py::test_join_s3 12.63s call test_keeper_session/test.py::test_session_close_shutdown 12.21s call test_keeper_back_to_back/test.py::test_watchers 11.66s setup test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size 11.13s teardown test_keeper_reconfig_replace_leader_in_one_command/test.py::test_reconfig_replace_leader_in_one_command 10.72s call test_memory_limit_observer/test.py::test_observe_memory_limit 8.96s setup test_keeper_reconfig_replace_leader_in_one_command/test.py::test_reconfig_replace_leader_in_one_command 8.64s call test_modify_engine_on_restart/test_unusual_path.py::test_modify_engine_on_restart_with_unusual_path 8.57s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 8.46s teardown test_part_uuid/test.py::test_part_uuid 8.34s call test_join_set_family_s3/test.py::test_set_s3 7.86s call test_multi_access_storage_role_management/test.py::test_role_from_different_storages 7.54s call test_mask_sensitive_info/test.py::test_encryption_functions 7.47s call test_recovery_replica/test.py::test_choose_source_replica 6.75s teardown test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 6.53s call test_recovery_replica/test.py::test_recovery 6.46s teardown test_mask_sensitive_info/test.py::test_table_functions 6.43s call test_kafka_bad_messages/test.py::test_bad_messages_parsing_exception 6.30s teardown test_recovery_replica/test.py::test_recovery 6.14s teardown test_keeper_back_to_back/test.py::test_watchers 5.94s teardown test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 5.93s call test_filesystem_cache/test.py::test_system_sync_filesystem_cache 5.52s teardown test_intersecting_parts/test.py::test_intersect_parts_when_restart 5.49s call test_read_temporary_tables_on_failure/test.py::test_different_versions 5.39s call test_mask_sensitive_info/test.py::test_backup_to_s3 5.04s call test_multi_access_storage_role_management/test.py::test_quotas 4.93s teardown test_filesystem_cache/test.py::test_system_sync_filesystem_cache 4.89s call test_multi_access_storage_role_management/test.py::test_settings_profiles 4.88s teardown test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 4.83s teardown test_memory_limit_observer/test.py::test_observe_memory_limit 4.74s call test_multi_access_storage_role_management/test.py::test_roles 4.73s call test_mask_sensitive_info/test.py::test_create_alter_user 4.49s call test_part_uuid/test.py::test_part_uuid 4.44s call test_multi_access_storage_role_management/test.py::test_users 4.38s teardown test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 4.25s call test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 4.25s teardown test_read_temporary_tables_on_failure/test.py::test_different_versions 4.25s call test_mask_sensitive_info/test.py::test_table_function_ways_to_call 4.21s teardown test_http_native/test.py::test_http_native_returns_timezone 3.95s teardown test_logs_level/test.py::test_check_client_logs_level 3.67s teardown test_keeper_client/test.py::test_set_without_version 3.59s call test_mask_sensitive_info/test.py::test_create_database 3.55s teardown test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch 3.03s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 2.99s call test_intersecting_parts/test.py::test_intersect_parts_when_restart 2.84s teardown test_modify_engine_on_restart/test_unusual_path.py::test_modify_engine_on_restart_with_unusual_path 2.79s call test_mask_sensitive_info/test.py::test_create_dictionary 2.78s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 2.78s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 2.77s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 2.67s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 2.65s teardown test_keeper_four_word_command/test_allow_list.py::test_allow_list 2.53s call test_log_family_s3/test.py::test_log_family_s3[TinyLog] 2.50s teardown test_keeper_nodes_move/test.py::test_node_move 2.47s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 2.44s call test_mask_sensitive_info/test.py::test_on_cluster 2.44s teardown test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 2.41s teardown test_keeper_four_word_command/test.py::test_cmd_ydld 2.41s teardown test_keeper_session/test.py::test_session_timeout 2.34s teardown test_keeper_mntr_pressure/test.py::test_aggressive_mntr 2.20s call test_log_family_s3/test.py::test_log_family_s3[Log] 2.17s call test_log_family_s3/test.py::test_log_family_s3[StripeLog] 2.13s teardown test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 2.01s teardown test_multi_access_storage_role_management/test.py::test_users 1.74s call test_keeper_four_word_command/test_allow_list.py::test_allow_list 1.68s teardown test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 1.24s call test_mask_sensitive_info/test.py::test_backup_table 1.19s call test_filesystem_cache/test.py::test_caches_with_the_same_configuration_2[node_caches_with_same_path] 1.04s call test_keeper_four_word_command/test.py::test_cmd_ydld 1.04s call test_keeper_four_word_command/test.py::test_cmd_rqld 0.97s call test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 0.85s call test_merge_table_over_distributed/test.py::test_filtering 0.84s call test_keeper_four_word_command/test.py::test_cmd_mntr 0.77s call test_keeper_back_to_back/test.py::test_stats 0.65s call test_keeper_four_word_command/test.py::test_cmd_lgif 0.58s call test_keeper_back_to_back/test.py::test_end_of_watches_session 0.53s call test_merge_table_over_distributed/test.py::test_global_in 0.53s call test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 0.36s call test_keeper_four_word_command/test.py::test_cmd_crst 0.36s call test_keeper_four_word_command/test.py::test_cmd_stat 0.35s call test_http_native/test.py::test_http_native_returns_timezone 0.34s call test_keeper_four_word_command/test.py::test_cmd_wchc 0.33s call test_keeper_four_word_command/test.py::test_cmd_wchs 0.30s call test_keeper_four_word_command/test.py::test_cmd_cons 0.30s call test_keeper_four_word_command/test.py::test_cmd_dump 0.30s call test_keeper_four_word_command/test.py::test_cmd_srvr 0.27s call test_keeper_four_word_command/test.py::test_cmd_wchp 0.27s call test_keeper_four_word_command/test.py::test_cmd_clrs 0.26s call test_keeper_back_to_back/test.py::test_end_of_session 0.22s call test_keeper_back_to_back/test.py::test_sequential_nodes 0.22s call test_logs_level/test.py::test_check_client_logs_level 0.16s call test_keeper_four_word_command/test.py::test_cmd_srst 0.16s call test_keeper_back_to_back/test.py::test_multitransactions 0.13s call test_keeper_back_to_back/test.py::test_simple_commands 0.12s call test_keeper_four_word_command/test.py::test_cmd_csnp 0.11s call test_keeper_four_word_command/test.py::test_cmd_conf 0.08s setup test_keeper_client/test.py::test_rm_with_version 0.08s setup test_keeper_client/test.py::test_set_without_version 0.08s setup test_keeper_client/test.py::test_set_with_version 0.08s setup test_keeper_client/test.py::test_rm_without_version 0.08s setup test_keeper_client/test.py::test_quoted_argument_parsing 0.07s setup test_keeper_client/test.py::test_find_super_nodes 0.07s setup test_keeper_client/test.py::test_big_family 0.07s setup test_keeper_client/test.py::test_four_letter_word_commands 0.07s setup test_keeper_client/test.py::test_get_all_children_number 0.06s setup test_keeper_client/test.py::test_delete_stale_backups 0.05s call test_keeper_client/test.py::test_find_super_nodes 0.04s call test_keeper_client/test.py::test_big_family 0.04s call test_keeper_client/test.py::test_get_all_children_number 0.04s call test_keeper_client/test.py::test_delete_stale_backups 0.03s call test_keeper_client/test.py::test_quoted_argument_parsing 0.03s call test_keeper_session/test.py::test_session_timeout 0.03s teardown test_keeper_back_to_back/test.py::test_random_requests 0.02s teardown test_keeper_client/test.py::test_set_with_version 0.02s teardown test_keeper_client/test.py::test_rm_with_version 0.02s teardown test_keeper_client/test.py::test_find_super_nodes 0.02s teardown test_keeper_client/test.py::test_delete_stale_backups 0.02s teardown test_keeper_client/test.py::test_get_all_children_number 0.02s teardown test_keeper_client/test.py::test_four_letter_word_commands 0.02s teardown test_keeper_client/test.py::test_rm_without_version 0.02s call test_keeper_client/test.py::test_base_commands 0.02s teardown test_keeper_client/test.py::test_quoted_argument_parsing 0.02s teardown test_keeper_client/test.py::test_base_commands 0.01s call test_keeper_client/test.py::test_set_with_version 0.01s teardown test_keeper_client/test.py::test_big_family 0.01s call test_keeper_client/test.py::test_rm_with_version 0.01s call test_keeper_four_word_command/test.py::test_cmd_isro 0.01s call test_keeper_four_word_command/test.py::test_cmd_ruok 0.01s call test_keeper_client/test.py::test_set_without_version 0.01s teardown test_keeper_back_to_back/test.py::test_concurrent_watches 0.01s call test_keeper_client/test.py::test_rm_without_version 0.00s teardown test_kafka_bad_messages/test.py::test_bad_messages_parsing_stream 0.00s call test_keeper_client/test.py::test_four_letter_word_commands 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[Log] 0.00s setup test_mask_sensitive_info/test.py::test_table_function_ways_to_call 0.00s teardown test_join_set_family_s3/test.py::test_join_s3 0.00s teardown test_mask_sensitive_info/test.py::test_create_database 0.00s teardown test_jbod_balancer/test.py::test_jbod_balanced_merge 0.00s teardown test_kafka_bad_messages/test.py::test_bad_messages_parsing_exception 0.00s teardown test_keeper_back_to_back/test.py::test_end_of_watches_session 0.00s teardown test_keeper_back_to_back/test.py::test_stats 0.00s setup test_filesystem_cache/test.py::test_parallel_cache_loading_on_startup[node] 0.00s setup test_keeper_four_word_command/test.py::test_cmd_crst 0.00s setup test_kafka_bad_messages/test.py::test_bad_messages_parsing_stream 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 0.00s setup test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s teardown test_mask_sensitive_info/test.py::test_backup_table 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s teardown test_recovery_replica/test.py::test_choose_source_replica 0.00s setup test_keeper_back_to_back/test.py::test_watchers 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 0.00s teardown test_filesystem_cache/test.py::test_caches_with_the_same_configuration_2[node_caches_with_same_path] 0.00s setup test_mask_sensitive_info/test.py::test_create_database 0.00s setup test_keeper_four_word_command/test.py::test_cmd_stat 0.00s setup test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch 0.00s setup test_mask_sensitive_info/test.py::test_create_alter_user 0.00s setup test_keeper_back_to_back/test.py::test_end_of_session 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_mntr 0.00s setup test_log_family_s3/test.py::test_log_family_s3[TinyLog] 0.00s setup test_keeper_four_word_command/test.py::test_cmd_cons 0.00s teardown test_keeper_session/test.py::test_session_close_shutdown 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_cons 0.00s setup test_multi_access_storage_role_management/test.py::test_users 0.00s setup test_keeper_back_to_back/test.py::test_multitransactions 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s setup test_keeper_back_to_back/test.py::test_sequential_nodes 0.00s setup test_multi_access_storage_role_management/test.py::test_settings_profiles 0.00s teardown test_multi_access_storage_role_management/test.py::test_quotas 0.00s teardown test_filesystem_cache/test.py::test_custom_cached_disk 0.00s setup test_kafka_bad_messages/test.py::test_bad_messages_to_mv 0.00s setup test_filesystem_cache/test.py::test_custom_cached_disk 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s teardown test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 0.00s setup test_mask_sensitive_info/test.py::test_backup_to_s3 0.00s teardown test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_lgif 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchc 0.00s teardown test_merge_table_over_distributed/test.py::test_filtering 0.00s setup test_recovery_replica/test.py::test_recovery 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_stat 0.00s setup test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 0.00s teardown test_keeper_back_to_back/test.py::test_multitransactions 0.00s setup test_mask_sensitive_info/test.py::test_encryption_functions 0.00s setup test_keeper_four_word_command/test.py::test_cmd_srst 0.00s setup test_keeper_back_to_back/test.py::test_random_requests 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_conf 0.00s setup test_mask_sensitive_info/test.py::test_create_dictionary 0.00s setup test_keeper_four_word_command/test.py::test_cmd_rqld 0.00s setup test_keeper_four_word_command/test.py::test_cmd_srvr 0.00s teardown test_keeper_back_to_back/test.py::test_end_of_session 0.00s teardown test_mask_sensitive_info/test.py::test_create_alter_user 0.00s setup test_keeper_four_word_command/test.py::test_cmd_isro 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_clrs 0.00s teardown test_multi_access_storage_role_management/test.py::test_settings_profiles 0.00s teardown test_multi_access_storage_role_management/test.py::test_roles 0.00s setup test_keeper_four_word_command/test.py::test_cmd_dump 0.00s setup test_mask_sensitive_info/test.py::test_on_cluster 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srvr 0.00s setup test_keeper_four_word_command/test.py::test_cmd_lgif 0.00s teardown test_keeper_back_to_back/test.py::test_sequential_nodes 0.00s setup test_keeper_back_to_back/test.py::test_end_of_watches_session 0.00s teardown test_mask_sensitive_info/test.py::test_backup_to_s3 0.00s setup test_keeper_four_word_command/test.py::test_cmd_conf 0.00s setup test_join_set_family_s3/test.py::test_set_s3 0.00s setup test_multi_access_storage_role_management/test.py::test_role_from_different_storages 0.00s teardown test_mask_sensitive_info/test.py::test_encryption_functions 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_dump 0.00s setup test_filesystem_cache/test.py::test_force_filesystem_cache_on_merges 0.00s setup test_memory_limit_observer/test.py::test_observe_memory_limit 0.00s setup test_keeper_session/test.py::test_session_timeout 0.00s setup test_filesystem_cache/test.py::test_system_sync_filesystem_cache 0.00s teardown test_mask_sensitive_info/test.py::test_create_table 0.00s teardown test_filesystem_cache/test.py::test_parallel_cache_loading_on_startup[node] 0.00s teardown test_filesystem_cache/test.py::test_force_filesystem_cache_on_merges 0.00s setup test_multi_access_storage_role_management/test.py::test_roles 0.00s setup test_keeper_four_word_command/test.py::test_cmd_mntr 0.00s setup test_keeper_four_word_command/test.py::test_cmd_ydld 0.00s setup test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchp 0.00s teardown test_multi_access_storage_role_management/test.py::test_role_from_different_storages 0.00s setup test_mask_sensitive_info/test.py::test_create_table 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.00s setup test_keeper_back_to_back/test.py::test_simple_commands 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srst 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchs 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_crst 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s setup test_keeper_back_to_back/test.py::test_stats 0.00s setup test_merge_table_over_distributed/test.py::test_global_in 0.00s setup test_keeper_four_word_command/test.py::test_cmd_csnp 0.00s setup test_mask_sensitive_info/test.py::test_table_functions 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s setup test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchp 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_csnp 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchc 0.00s teardown test_keeper_back_to_back/test.py::test_simple_commands 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchs 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_rqld 0.00s teardown test_mask_sensitive_info/test.py::test_on_cluster 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_isro 0.00s teardown test_mask_sensitive_info/test.py::test_table_function_ways_to_call 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s teardown test_mask_sensitive_info/test.py::test_create_dictionary 0.00s teardown test_merge_table_over_distributed/test.py::test_global_in =========================== short test summary info ============================ ERROR test_keeper_force_recovery_single_node/test.py::test_cluster_recovery PASSED test_merge_table_over_distributed/test.py::test_filtering PASSED test_merge_table_over_distributed/test.py::test_global_in PASSED test_keeper_four_word_command/test.py::test_cmd_clrs PASSED test_keeper_four_word_command/test.py::test_cmd_conf PASSED test_filesystem_cache/test.py::test_caches_with_the_same_configuration_2[node_caches_with_same_path] PASSED test_keeper_four_word_command/test.py::test_cmd_cons PASSED test_keeper_four_word_command/test.py::test_cmd_crst PASSED test_keeper_four_word_command/test.py::test_cmd_csnp PASSED test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed PASSED test_keeper_four_word_command/test.py::test_cmd_dump PASSED test_keeper_four_word_command/test.py::test_cmd_isro PASSED test_keeper_four_word_command/test.py::test_cmd_lgif PASSED test_log_family_s3/test.py::test_log_family_s3[Log] PASSED test_keeper_client/test.py::test_base_commands PASSED test_keeper_client/test.py::test_big_family PASSED test_keeper_four_word_command/test.py::test_cmd_mntr PASSED test_keeper_client/test.py::test_delete_stale_backups PASSED test_keeper_client/test.py::test_find_super_nodes PASSED test_keeper_client/test.py::test_four_letter_word_commands PASSED test_keeper_client/test.py::test_get_all_children_number PASSED test_keeper_client/test.py::test_quoted_argument_parsing PASSED test_keeper_client/test.py::test_rm_with_version PASSED test_keeper_client/test.py::test_rm_without_version PASSED test_keeper_client/test.py::test_set_with_version PASSED test_keeper_client/test.py::test_set_without_version PASSED test_keeper_four_word_command/test.py::test_cmd_rqld PASSED test_keeper_four_word_command/test.py::test_cmd_ruok PASSED test_keeper_four_word_command/test.py::test_cmd_srst PASSED test_multi_access_storage_role_management/test.py::test_quotas PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] PASSED test_keeper_four_word_command/test.py::test_cmd_srvr PASSED test_keeper_four_word_command/test.py::test_cmd_stat PASSED test_keeper_four_word_command/test.py::test_cmd_wchc PASSED test_mask_sensitive_info/test.py::test_backup_table PASSED test_keeper_four_word_command/test.py::test_cmd_wchp PASSED test_keeper_four_word_command/test.py::test_cmd_wchs PASSED test_keeper_four_word_command/test.py::test_cmd_ydld PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] PASSED test_mask_sensitive_info/test.py::test_backup_to_s3 PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] PASSED test_multi_access_storage_role_management/test.py::test_role_from_different_storages PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] PASSED test_mask_sensitive_info/test.py::test_create_alter_user PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] PASSED test_multi_access_storage_role_management/test.py::test_roles PASSED test_mask_sensitive_info/test.py::test_create_database PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] PASSED test_filesystem_cache/test.py::test_custom_cached_disk PASSED test_multi_access_storage_role_management/test.py::test_settings_profiles PASSED test_mask_sensitive_info/test.py::test_create_dictionary PASSED test_kafka_bad_messages/test.py::test_bad_messages_parsing_exception PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] PASSED test_multi_access_storage_role_management/test.py::test_users PASSED test_keeper_back_to_back/test.py::test_concurrent_watches PASSED test_keeper_back_to_back/test.py::test_end_of_session PASSED test_keeper_back_to_back/test.py::test_end_of_watches_session PASSED test_keeper_back_to_back/test.py::test_multitransactions PASSED test_keeper_session/test.py::test_session_close_shutdown PASSED test_keeper_session/test.py::test_session_timeout PASSED test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size PASSED test_http_native/test.py::test_http_native_returns_timezone PASSED test_mask_sensitive_info/test.py::test_create_table PASSED test_memory_limit_observer/test.py::test_observe_memory_limit PASSED test_intersecting_parts/test.py::test_intersect_parts_when_restart PASSED test_mask_sensitive_info/test.py::test_encryption_functions PASSED test_jbod_balancer/test.py::test_jbod_balanced_merge PASSED test_mask_sensitive_info/test.py::test_on_cluster PASSED test_join_set_family_s3/test.py::test_join_s3 PASSED test_mask_sensitive_info/test.py::test_table_function_ways_to_call PASSED test_keeper_back_to_back/test.py::test_random_requests PASSED test_keeper_back_to_back/test.py::test_sequential_nodes PASSED test_keeper_back_to_back/test.py::test_simple_commands PASSED test_keeper_back_to_back/test.py::test_stats PASSED test_join_set_family_s3/test.py::test_set_s3 PASSED test_keeper_back_to_back/test.py::test_watchers PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list PASSED test_recovery_replica/test.py::test_choose_source_replica PASSED test_recovery_replica/test.py::test_recovery PASSED test_mask_sensitive_info/test.py::test_table_functions PASSED test_logs_level/test.py::test_check_client_logs_level PASSED test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch PASSED test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery PASSED test_filesystem_cache/test.py::test_force_filesystem_cache_on_merges PASSED test_keeper_reconfig_replace_leader_in_one_command/test.py::test_reconfig_replace_leader_in_one_command PASSED test_keeper_nodes_move/test.py::test_node_move PASSED test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task PASSED test_filesystem_cache/test.py::test_parallel_cache_loading_on_startup[node] PASSED test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions PASSED test_filesystem_cache/test.py::test_system_sync_filesystem_cache PASSED test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table PASSED test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 PASSED test_modify_engine_on_restart/test_unusual_path.py::test_modify_engine_on_restart_with_unusual_path PASSED test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery PASSED test_part_uuid/test.py::test_part_uuid PASSED test_kafka_bad_messages/test.py::test_bad_messages_parsing_stream PASSED test_kafka_bad_messages/test.py::test_bad_messages_to_mv PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr =================== 100 passed, 1 error in 433.25s (0:07:13) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 437, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_u7fdr2 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=2993bc2bf171 -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_filesystem_cache/test.py::test_caches_with_the_same_configuration_2[node_caches_with_same_path]' test_filesystem_cache/test.py::test_custom_cached_disk test_filesystem_cache/test.py::test_force_filesystem_cache_on_merges 'test_filesystem_cache/test.py::test_parallel_cache_loading_on_startup[node]' test_filesystem_cache/test.py::test_system_sync_filesystem_cache test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 test_http_native/test.py::test_http_native_returns_timezone test_intersecting_parts/test.py::test_intersect_parts_when_restart test_jbod_balancer/test.py::test_jbod_balanced_merge test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch test_join_set_family_s3/test.py::test_join_s3 test_join_set_family_s3/test.py::test_set_s3 test_kafka_bad_messages/test.py::test_bad_messages_parsing_exception test_kafka_bad_messages/test.py::test_bad_messages_parsing_stream test_kafka_bad_messages/test.py::test_bad_messages_to_mv test_keeper_back_to_back/test.py::test_concurrent_watches test_keeper_back_to_back/test.py::test_end_of_session test_keeper_back_to_back/test.py::test_end_of_watches_session test_keeper_back_to_back/test.py::test_multitransactions test_keeper_back_to_back/test.py::test_random_requests test_keeper_back_to_back/test.py::test_sequential_nodes test_keeper_back_to_back/test.py::test_simple_commands test_keeper_back_to_back/test.py::test_stats test_keeper_back_to_back/test.py::test_watchers test_keeper_client/test.py::test_base_commands test_keeper_client/test.py::test_big_family test_keeper_client/test.py::test_delete_stale_backups test_keeper_client/test.py::test_find_super_nodes test_keeper_client/test.py::test_four_letter_word_commands test_keeper_client/test.py::test_get_all_children_number test_keeper_client/test.py::test_quoted_argument_parsing test_keeper_client/test.py::test_rm_with_version test_keeper_client/test.py::test_rm_without_version test_keeper_client/test.py::test_set_with_version test_keeper_client/test.py::test_set_without_version test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test.py::test_cmd_ydld test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_reconfig_replace_leader_in_one_command/test.py::test_reconfig_replace_leader_in_one_command test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_logs_level/test.py::test_check_client_logs_level test_mask_sensitive_info/test.py::test_backup_table test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size test_memory_limit_observer/test.py::test_observe_memory_limit test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed test_modify_engine_on_restart/test_unusual_path.py::test_modify_engine_on_restart_with_unusual_path test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 ' returned non-zero exit status 1.